Bayesian data analysis - BRMS demos
Aki Vehtari
First version 2023-12-05. Last modified 2024-01-17.
Setup
Load packages
library(tidyr)
library(dplyr)
library(tibble)
library(pillar)
library(stringr)
library(brms)
options(brms.backend = "cmdstanr", mc.cores = 2)
library(posterior)
options(pillar.negative = FALSE)
library(loo)
library(priorsense)
library(ggplot2)
library(bayesplot)
theme_set(bayesplot::theme_default(base_family = "sans"))
library(tidybayes)
library(ggdist)
library(patchwork)
library(RColorBrewer)
SEED <- 48927 # set random seed for reproducability1 Introduction
This notebook contains several examples of how to use Stan in R with brms. This notebook assumes basic knowledge of Bayesian inference and MCMC. The examples are related to Bayesian data analysis course.
2 Bernoulli model
Toy data with sequence of failures (0) and successes (1). We would like to learn about the unknown probability of success.
data_bern <- data.frame(y = c(1, 1, 1, 0, 1, 1, 1, 0, 1, 0))As usual in case of generalizd linear models, (GLMs) brms defines the priors on the latent model parameters. With Bernoulli the default link function is logit, and thus the prior is set on logit(theta). As there are no covariates logit(theta)=Intercept. The brms default prior for Intercept is student_t(3, 0, 2.5), but we use student_t(7, 0, 1.5) which is close to logistic distribution, and thus makes the prior near-uniform for theta. We can simulate from these priors to check the implied prior on theta. We next compare the result to using normal(0, 1) prior on logit probability. We visualize the implied priors by sampling from the priors.
data.frame(theta = plogis(ggdist::rstudent_t(n=20000, df=3, mu=0, sigma=2.5))) |>
mcmc_hist() +
xlim(c(0,1)) +
labs(title='Default brms student_t(3, 0, 2.5) prior on Intercept')
data.frame(theta = plogis(ggdist::rstudent_t(n=20000, df=7, mu=0, sigma=1.5))) |>
mcmc_hist() +
xlim(c(0,1)) +
labs(title='student_t(7, 0, 1.5) prior on Intercept')
Almost uniform prior on theta could be obtained also with normal(0,1.5)
data.frame(theta = plogis(rnorm(n=20000, mean=0, sd=1.5))) |>
mcmc_hist() +
xlim(c(0,1)) +
labs(title='normal(0, 1.5) prior on Intercept')
Formula y ~ 1 corresponds to a model $() =
#\alpha\times 1 = \alpha$. `brms? denotes the $\alpha$ as `Intercept`.fit_bern <- brm(y ~ 1, family = bernoulli(), data = data_bern,
prior = prior(student_t(7, 0, 1.5), class='Intercept'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics.
fit_bern Family: bernoulli
Links: mu = logit
Formula: y ~ 1
Data: data_bern (Number of observations: 10)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 0.76 0.64 -0.43 2.09 1.00 1734 1726
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Extract the posterior draws
draws <- as_draws_df(fit_bern)We can get summary information using summarise_draws()
draws |>
subset_draws(variable='b_Intercept') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 b_Intercept 0.763 0.746 0.641 0.636 -0.242 1.90 1.00 1734. 1726.We can compute the probability of success by using plogis which is equal to inverse-logit function
draws <- draws |>
mutate_variables(theta=plogis(b_Intercept))Summary of theta by using summarise_draws()
draws |>
subset_draws(variable='theta') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 theta 0.668 0.678 0.130 0.134 0.440 0.870 1.00 1734. 1726.Histogram of theta
mcmc_hist(draws, pars='theta') +
xlab('theta') +
xlim(c(0,1))
Prior and likelihood sensitivity plot shows posterior density estimate depending on amount of power-scaling. Overlapping line indicate low sensitivity and wider gaps between line indicate greater sensitivity.
theta <- draws |>
subset_draws(variable='theta')
powerscale_sequence(fit_bern, prediction = \(x, ...) theta) |>
powerscale_plot_dens(variables='theta') +
# switch rows and cols
facet_grid(rows=vars(.data$variable),
cols=vars(.data$component)) +
# cleaning
ggtitle(NULL,NULL) +
labs(x='theta', y=NULL) +
scale_y_continuous(breaks=NULL) +
theme(axis.line.y=element_blank(),
strip.text.y=element_blank()) +
xlim(c(0,1))
We can summarise the prior and likelihood sensitivity using cumulative Jensen-Shannon distance.
powerscale_sensitivity(fit_bern, prediction = \(x, ...) theta)$sensitivity |>
filter(variable=='theta') |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 1 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 theta 0.04 0.11 - 3 Binomial model
Instead of sequence of 0’s and 1’s, we can summarize the data with the number of trials and the number successes and use Binomial model. The prior is specified in the ‘latent space’. The actual probability of success, theta = plogis(alpha), where plogis is the inverse of the logistic function.
Binomial model with the same data and prior
data_bin <- data.frame(N = c(10), y = c(7))Formula y | trials(N) ~ 1 corresponds to a model \(\mathrm{logit}(\theta) = \alpha\), and the number of trials for each observation is provided by | trials(N)
fit_bin <- brm(y | trials(N) ~ 1, family = binomial(), data = data_bin,
prior = prior(student_t(7, 0,1.5), class='Intercept'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics.
fit_bin Family: binomial
Links: mu = logit
Formula: y | trials(N) ~ 1
Data: data_bin (Number of observations: 1)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 0.75 0.64 -0.47 2.07 1.00 1699 1508
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).The diagnostic indicates prior-data conflict, that is, both prior and likelihood are informative. If there is true strong prior information that would justify the normal(0,1) prior, then this is fine, but otherwise more thinking is required (goal is not adjust prior to remove diagnostic warnings withoyt thinking). In this toy example, we proceed with this prior.
Extract the posterior draws
draws <- as_draws_df(fit_bin)We can get summary information using summarise_draws()
draws |>
subset_draws(variable='b_Intercept') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 b_Intercept 0.749 0.741 0.635 0.620 -0.266 1.87 1.00 1699. 1508.We can compute the probability of success by using plogis which is equal to inverse-logit function
draws <- draws |>
mutate_variables(theta=plogis(b_Intercept))Summary of theta by using summarise_draws()
draws |>
subset_draws(variable='theta') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 theta 0.665 0.677 0.130 0.131 0.434 0.866 1.00 1699. 1508.Histogram of theta
mcmc_hist(draws, pars='theta') +
xlab('theta') +
xlim(c(0,1))
Re-run the model with a new data dataset without recompiling
data_bin <- data.frame(N = c(5), y = c(4))
fit_bin <- update(fit_bin, newdata = data_bin)Check the summary of the posterior and inference diagnostics.
fit_bin Family: binomial
Links: mu = logit
Formula: y | trials(N) ~ 1
Data: data_bin (Number of observations: 1)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 1.05 0.92 -0.61 2.98 1.00 1470 2036
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Extract the posterior draws
draws <- as_draws_df(fit_bin)We can get summary information using summarise_draws()
draws |>
subset_draws(variable='b_Intercept') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 b_Intercept 1.05 1.02 0.920 0.890 -0.399 2.65 1.00 1470. 2036.We can compute the probability of success by using plogis which is equal to inverse-logit function
draws <- draws |>
mutate_variables(theta=plogis(b_Intercept))Summary of theta by using summarise_draws()
draws |>
subset_draws(variable='theta') |>
summarise_draws()# A tibble: 1 × 10
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 theta 0.710 0.735 0.162 0.164 0.402 0.934 1.00 1470. 2036.Histogram of theta
mcmc_hist(draws, pars='theta') +
xlab('theta') +
xlim(c(0,1))
4 Comparison of two groups with Binomial
An experiment was performed to estimate the effect of beta-blockers on mortality of cardiac patients. A group of patients were randomly assigned to treatment and control groups:
- out of 674 patients receiving the control, 39 died
- out of 680 receiving the treatment, 22 died
Data, where grp2 is an indicator variable defined as a factor type, which is useful for categorical variables.
data_bin2 <- data.frame(N = c(674, 680),
y = c(39,22),
grp2 = factor(c('control','treatment')))To analyse whether the treatment is useful, we can use Binomial model for both groups and compute odds-ratio. To recreate the model as two independent (separate) binomial models, we use formula y | trials(N) ~ 0 + grp2, which corresponds to a model \(\mathrm{logit}(\theta) = \alpha \times 0 + \beta_\mathrm{control}\times x_\mathrm{control} + \beta_\mathrm{treatment}\times x_\mathrm{treatment} = \beta_\mathrm{control}\times x_\mathrm{control} + \beta_\mathrm{treatment}\times x_\mathrm{treatment}\), where \(x_\mathrm{control}\) is a vector with 1 for control and 0 for treatment, and \(x_\mathrm{treatemnt}\) is a vector with 1 for treatemnt and 0 for control. As only of the vectors have 1, this corresponds to separate models \(\mathrm{logit}(\theta_\mathrm{control}) = \beta_\mathrm{control}\) and \(\mathrm{logit}(\theta_\mathrm{treatment}) = \beta_\mathrm{treatment}\). We can provide the same prior for all \(\beta\)’s by setting the prior with class='b'. With prior student_t(7, 0,1.5), both \(\beta\)’s are shrunk towards 0, but independently.
fit_bin2 <- brm(y | trials(N) ~ 0 + grp2, family = binomial(), data = data_bin2,
prior = prior(student_t(7, 0,1.5), class='b'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics. brms is using the first factor level control as the baseline and thus reports the coefficient (population-level effect) for treatment (shown s grp2treatment) Check the summary of the posterior and inference diagnostics. With ~ 0 + grp2 there is no Intercept and and are presented as grp2control and grp2treatment.
fit_bin2 Family: binomial
Links: mu = logit
Formula: y | trials(N) ~ 0 + grp2
Data: data_bin2 (Number of observations: 2)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
grp2control -2.78 0.17 -3.13 -2.45 1.00 2406 2661
grp2treatment -3.38 0.21 -3.80 -2.99 1.00 3386 2390
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Compute theta for each group and the odds-ratio. brms uses bariable names b_grp2control and b_grp2treatment for \(\beta_\mathrm{control}\) and \(\beta_\mathrm{treatment}\) respectively.
draws_bin2 <- as_draws_df(fit_bin2) |>
mutate(theta_control = plogis(b_grp2control),
theta_treatment = plogis(b_grp2treatment),
oddsratio = (theta_treatment/(1-theta_treatment))/(theta_control/(1-theta_control)))Plot oddsratio
mcmc_hist(draws_bin2, pars='oddsratio') +
scale_x_continuous(breaks=seq(0.2,1.6,by=0.2))+
geom_vline(xintercept=1, linetype='dashed')
Probability that the oddsratio<1
draws_bin2 |>
mutate(poddsratio = oddsratio<1) |>
subset(variable='poddsratio') |>
summarise_draws(mean, mcse_mean)# A tibble: 1 × 3
variable mean mcse_mean
<chr> <dbl> <dbl>
1 poddsratio 0.984 0.00234oddsratio 95% posterior interval
draws_bin2 |>
subset(variable='oddsratio') |>
summarise_draws(~quantile(.x, probs = c(0.025, 0.975)), ~mcse_quantile(.x, probs = c(0.025, 0.975)))# A tibble: 1 × 5
variable `2.5%` `97.5%` mcse_q2.5 mcse_q97.5
<chr> <dbl> <dbl> <dbl> <dbl>
1 oddsratio 0.320 0.928 0.00381 0.0149Make prior sensitivity analysis by power-scaling both prior and likelihood. Focus on oddsratio which is the quantity of interest. We see that the likelihood is much more informative than the prior, and we would expect to see a different posterior only with a highly informative prior (possibly based on previous similar experiments).
oddsratio <- draws_bin2 |>
subset_draws(variable='oddsratio')Prior and likelihood sensitivity plot shows posterior density estimate depending on amount of power-scaling. Overlapping line indicate low sensitivity and wider gaps between line indicate greater sensitivity.
powerscale_sequence(fit_bin2, prediction = \(x, ...) oddsratio) |>
powerscale_plot_dens(variables='oddsratio') +
# switch rows and cols
facet_grid(rows=vars(.data$variable),
cols=vars(.data$component)) +
# cleaning
ggtitle(NULL,NULL) +
labs(x='Odds-ratio', y=NULL) +
scale_y_continuous(breaks=NULL) +
theme(axis.line.y=element_blank(),
strip.text.y=element_blank()) +
# reference line
geom_vline(xintercept=1, linetype='dashed')
We can summarise the prior and likelihood sensitivity using cumulative Jensen-Shannon distance.
powerscale_sensitivity(fit_bin2, prediction = \(x, ...) oddsratio, num_args=list(digits=2)
)$sensitivity |>
filter(variable=='oddsratio') |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 1 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 oddsratio 0.01 0.14 - Above we used formula y | trials(N) ~ 0 + grp2 to have separate model for control and treatment group. An alternative model y | trials(N) ~ grp2 which is equal to y | trials(N) ~ 1 + grp2, would correspond to a model $() = + x = + x. Now \(\alpha\) models the probability of death (via logistic link) in the control group and \(\alpha + \beta_\mathrm{treatment}\) models the probability of death (via logistic link) in the treatment group. Now the models for the groups are connected. Furthermore, if we set independent student_t(7, 0, 1.5) priors on \(\alpha\) and \(\beta_\mathrm{treatment}\), the implied priors on \(\theta_\mathrm{control}\) and \(\theta_\mathrm{treatment}\) are different. We can verify this with a prior simulation.
data.frame(theta_control = plogis(ggdist::rstudent_t(n=20000, df=7, mu=0, sigma=1.5))) |>
mcmc_hist() +
xlim(c(0,1)) +
labs(title='student_t(7, 0, 1.5) prior on Intercept') +
data.frame(theta_treatment = plogis(ggdist::rstudent_t(n=20000, df=7, mu=0, sigma=1.5))+
plogis(ggdist::rstudent_t(n=20000, df=7, mu=0, sigma=1.5))) |>
mcmc_hist() +
xlim(c(0,1)) +
labs(title='student_t(7, 0, 1.5) prior on Intercept and b_grp2treatment')
In this case, with relatively big treatment and control group, the likelihood is informative, and the difference between using y | trials(N) ~ 0 + grp2 or y | trials(N) ~ grp2 is negligible.
Third option would be a hierarchical model with formula y | trials(N) ~ 1 + (1 | grp2), which is equivalent to y | trials(N) ~ 1 + (1 | grp2), and corresponds to a model \(\mathrm{logit}(\theta) = \alpha \times 1 + \beta_\mathrm{control}\times x_\mathrm{control} + \beta_\mathrm{treatment}\times x_\mathrm{treatment}\), but now the prior on \(\beta_\mathrm{control}\) and \(\beta_\mathrm{treatment}\) is \(\mathrm{normal}(0, \sigma_\mathrm{grp})\). The default brms prior for \(\sigma_\mathrm{grp}\) is student_t(3, 0, 2.5). Now \(\alpha\) models the overall probablity of death (via logistic link), and \(\beta_\mathrm{control}\) and \(\beta_\mathrm{treatment}\) model the difference from that having the same prior. Prior for \(\beta_\mathrm{control}\) and \(\beta_\mathrm{treatment}\) includes unknown scale \(\sigma_\mathrm{grp}\). If the there is not difference between control and treatment groups, the posterior of \(\sigma_\mathrm{grp}\) has more mass near 0, and bigger the difference between control and treatment groups are, more mass there is away from 0. With just two groups, there is not much information about \(\sigma_\mathrm{grp}\), and unless there is a informative prior on \(\sigma_\mathrm{grp}\), two group hierarchical model is not that useful. Hierarchical models are more useful with more than two groups. In the following, we use the previously used student_t(7, 0,1.5) prior on intercept and the default brms prior student_t(3, 0, 2.5) on \(\sigma_\mathrm{grp}\).
fit_bin2 <- brm(y | trials(N) ~ 1 + (1 | grp2), family = binomial(), data = data_bin2,
prior = prior(student_t(7, 0,1.5), class='Intercept'),
seed = SEED, refresh = 0, control=list(adapt_delta=0.99))Check the summary of the posterior and inference diagnostics. The summary reports that there are Group-Level Effects: ~grp2 with 2 levels (control and treatment), with sd(Intercept) denoting \(\sigma_\mathrm{grp}\). In addition, the summary lists Population-Level Effects: Intercept (\(\alpha\)) as in the prevous non-hierarchical models.
fit_bin2 Family: binomial
Links: mu = logit
Formula: y | trials(N) ~ 1 + (1 | grp2)
Data: data_bin2 (Number of observations: 2)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Group-Level Effects:
~grp2 (Number of levels: 2)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 1.64 1.45 0.15 5.69 1.00 549 932
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -2.21 1.24 -3.89 0.78 1.00 611 752
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).We can also look at the variable names brms uses internally
as_draws_rvars(fit_bin2)# A draws_rvars: 1000 iterations, 4 chains, and 5 variables
$b_Intercept: rvar<1000,4>[1] mean ± sd:
[1] -2.2 ± 1.2
$sd_grp2__Intercept: rvar<1000,4>[1] mean ± sd:
[1] 1.6 ± 1.4
$r_grp2: rvar<1000,4>[2,1] mean ± sd:
Intercept
control -0.6 ± 1.2
treatment -1.2 ± 1.3
$lprior: rvar<1000,4>[1] mean ± sd:
[1] -4.2 ± 0.71
$lp__: rvar<1000,4>[1] mean ± sd:
[1] -13 ± 1.8 Although there is no difference, illustrate how to compute the oddsratio from hierarchical model
draws_bin2 <- as_draws_df(fit_bin2)
oddsratio <- draws_bin2 |>
mutate_variables(theta_control = plogis(b_Intercept + `r_grp2[control,Intercept]`),
theta_treatment = plogis(b_Intercept + `r_grp2[treatment,Intercept]`),
oddsratio = (theta_treatment/(1-theta_treatment))/(theta_control/(1-theta_control))) |>
subset_draws(variable='oddsratio')
oddsratio |> mcmc_hist() +
scale_x_continuous(breaks=seq(0.2,1.6,by=0.2))+
geom_vline(xintercept=1, linetype='dashed')
Make also prior sensitivity analysis with focus on oddsratio.
powerscale_sensitivity(fit_bin2, prediction = \(x, ...) oddsratio, num_args=list(digits=2)
)$sensitivity |>
filter(variable=='oddsratio') |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 1 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 oddsratio 0.01 0.14 - 5 Linear Gaussian model
Use the Kilpisjärvi summer month temperatures 1952–2022 data from aaltobda package
load(url('https://github.com/avehtari/BDA_course_Aalto/raw/master/rpackage/data/kilpisjarvi2022.rda'))
data_lin <- data.frame(year = kilpisjarvi2022$year,
temp = kilpisjarvi2022$temp.summer)Plot the data
data_lin |>
ggplot(aes(year, temp)) +
geom_point(color=2) +
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
guides(linetype = "none")
To analyse has there been change in the average summer month temperature we use a linear model with Gaussian model for the unexplained variation. By default brms uses uniform prior for the coefficients.
Formula temp ~ year corresponds to model \(\mathrm{temp} ~ \mathrm{normal}(\alpha + \beta \times \mathrm{temp}, \sigma). The model could also be defined as `temp ~ 1 + year` which explicitly shows the intercept (\)$) part. Using the variable names brms uses the model can be written also as temp ~ normal(b_Intercept*1 + b_year*year, sigma). We start with the default priors to see some tricks that brms does behind the curtain.
fit_lin <- brm(temp ~ year, data = data_lin, family = gaussian(),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics.
fit_lin Family: gaussian
Links: mu = identity; sigma = identity
Formula: temp ~ year
Data: data_lin (Number of observations: 71)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -34.69 12.49 -58.73 -10.19 1.00 3995 3035
year 0.02 0.01 0.01 0.03 1.00 3996 3035
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 1.08 0.09 0.91 1.28 1.00 3057 3011
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Convergence diagnostics look good. We see that posterior mean of Intercept is -34.7, which may sound strange, but that is the intercept at year 0, that is, very far from the data range, and thus doesn’t have meaningful interpretation directly. The posterior mean of year coefficient is 0.02, that is, we estimate that the summer temperature is increasing 0.02°C per year (which would make 1°C in 50 years).
We can check \(R^2\) which corresponds to the proporion of variance explained by the model. The linear model explains 0.16=16% of the total data variance.
bayes_R2(fit_lin) |> round(2) Estimate Est.Error Q2.5 Q97.5
R2 0.16 0.07 0.03 0.3We can check the all the priors used.
prior_summary(fit_lin) prior class coef group resp dpar nlpar lb ub source
(flat) b default
(flat) b year (vectorized)
student_t(3, 9.5, 2.5) Intercept default
student_t(3, 0, 2.5) sigma 0 defaultWe see that class=b and coef=year have flat, that is, improper uniform prior, Intercept has student_t(3, 9.5, 2.5), and sigma has student_t(3, 0, 2.5) prior. In general it is good to use proper priors, but sometimes flat priors are fine and produce proper posterior (like in this case). Important part here is that by default, brms sets the prior on Intercept after centering the covariate values (design matrix). In this case, brms uses temp - mean(temp) = temp - 1987 instead of original years. This in general improves the sampling efficiency. As the Intercept is now defined at the middle of the data, the default Intercept prior is centered on median of the target (here target is year). If we would like to set informative priors, we need to set the informative prior on Intercept given the centered covariate values. We can turn of the centering by setting argument center=FALSE, and we can set the prior on original intercept by using a formula temp ~ 0 + Intercept + year. In this case, we are happy with the default prior for the intercept. In this specific casse, the flat prior on coefficient is also fine, but we add an weakly informative prior just for the illustration. Let’s assume we expect the temperature to change less than 1°C in 10 years. With student_t(3, 0, 0.03) about 95% prior mass has less than 0.1°C change in year, and with low degrees of freedom (3) we have thick tails making the likelihood dominate in case of prior-data conflict. In real life, we do have much more information about the temperature change, and naturally a hierarchical spatio-temporal model with all temperature measurement locations would be even better.
fit_lin <- brm(temp ~ year, data = data_lin, family = gaussian(),
prior = prior(student_t(3, 0, 0.03), class='b'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics.
fit_lin Family: gaussian
Links: mu = identity; sigma = identity
Formula: temp ~ year
Data: data_lin (Number of observations: 71)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -32.54 12.28 -56.70 -9.01 1.00 4183 3259
year 0.02 0.01 0.01 0.03 1.00 4182 3259
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 1.08 0.09 0.92 1.27 1.00 3494 2709
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Make prior sensitivity analysis by power-scaling both prior and likelihood.
powerscale_sensitivity(fit_lin)$sensitivity |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 3 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 b_Intercept 0.03 0.09 -
2 b_year 0.03 0.09 -
3 sigma 0.00 0.13 - Our weakly informative proper prior has negligible sensitivity, and the likelihood is informative. Extract the posterior draws and check the summaries
draws_lin <- as_draws_df(fit_lin)
draws_lin |> summarise_draws()# A tibble: 5 × 10
variable mean median sd mad q5 q95 rhat ess_bulk
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 b_Intercept -3.25e+1 -3.24e+1 1.23e+1 1.24e+1 -5.29e+1 -1.29e+1 1.00 4183.
2 b_year 2.11e-2 2.11e-2 6.18e-3 6.22e-3 1.12e-2 3.14e-2 1.00 4182.
3 sigma 1.08e+0 1.07e+0 9.14e-2 9.08e-2 9.43e-1 1.24e+0 1.00 3494.
4 lprior -1.08e+0 -1.06e+0 1.65e-1 1.65e-1 -1.38e+0 -8.51e-1 1.00 4173.
5 lp__ -1.07e+2 -1.06e+2 1.21e+0 9.72e-1 -1.09e+2 -1.05e+2 1.00 1899.
# ℹ 1 more variable: ess_tail <dbl>If one of the columns is hidden we can force printing all columns
draws_lin |> summarise_draws() |> print(width=Inf)# A tibble: 5 × 10
variable mean median sd mad q5 q95 rhat
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 b_Intercept -32.5 -32.4 12.3 12.4 -52.9 -12.9 1.00
2 b_year 0.0211 0.0211 0.00618 0.00622 0.0112 0.0314 1.00
3 sigma 1.08 1.07 0.0914 0.0908 0.943 1.24 1.00
4 lprior -1.08 -1.06 0.165 0.165 -1.38 -0.851 1.00
5 lp__ -107. -106. 1.21 0.972 -109. -105. 1.00
ess_bulk ess_tail
<dbl> <dbl>
1 4183. 3259.
2 4182. 3259.
3 3494. 2709.
4 4173. 3285.
5 1899. 2576.Histogram of b_year
draws_lin |>
mcmc_hist(pars='b_year') +
xlab('Average temperature increase per year')
Probability that the coefficient b_year > 0 and the corresponding MCSE
draws_lin |>
mutate(I_b_year_gt_0 = b_year>0) |>
subset_draws(variable='I_b_year_gt_0') |>
summarise_draws(mean, mcse_mean)# A tibble: 1 × 3
variable mean mcse_mean
<chr> <dbl> <dbl>
1 I_b_year_gt_0 1 NAAll posterior draws have b_year>0, the probability gets rounded to 1, and MCSE is not available as the obserevd posterior variance is 0.
95% posterior interval for temperature increase per 100 years
draws_lin |>
mutate(b_year_100 = b_year*100) |>
subset_draws(variable='b_year_100') |>
summarise_draws(~quantile(.x, probs = c(0.025, 0.975)),
~mcse_quantile(.x, probs = c(0.025, 0.975)),
.num_args = list(digits = 2, notation = "dec"))# A tibble: 1 × 5
variable `2.5%` `97.5%` mcse_q2.5 mcse_q97.5
<chr> <dbl> <dbl> <dbl> <dbl>
1 b_year_100 0.93 3.33 0.03 0.03Plot posterior draws of the linear function values at each year. add_linpred_draws() takes the years from the data and uses fit_lin to make the predictions.
data_lin |>
add_linpred_draws(fit_lin) |>
# plot data
ggplot(aes(x=year, y=temp)) +
geom_point(color=2) +
# plot lineribbon for the linear model
stat_lineribbon(aes(y = .linpred), .width = c(.95), alpha = 1/2, color=brewer.pal(5, "Blues")[[5]]) +
# decoration
scale_fill_brewer()+
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
theme(legend.position="none")+
scale_x_continuous(breaks=seq(1950,2020,by=10))
Alternativelly plot a spaghetti plot for 100 draws
data_lin |>
add_linpred_draws(fit_lin, ndraws=100) |>
# plot data
ggplot(aes(x=year, y=temp)) +
geom_point(color=2) +
# plot a line for each posterior draw
geom_line(aes(y=.linpred, group=.draw), alpha = 1/2, color = brewer.pal(5, "Blues")[[3]])+
# decoration
scale_fill_brewer()+
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
theme(legend.position="none")+
scale_x_continuous(breaks=seq(1950,2020,by=10))
Plot posterior predictive distribution at each year until 2030 add_predicted_draws() takes the years from the data and uses fit_lin to make the predictions.
data_lin |>
add_row(year=2023:2030) |>
add_predicted_draws(fit_lin) |>
# plot data
ggplot(aes(x=year, y=temp)) +
geom_point(color=2) +
# plot lineribbon for the linear model
stat_lineribbon(aes(y = .prediction), .width = c(.95), alpha = 1/2, color=brewer.pal(5, "Blues")[[5]]) +
# decoration
scale_fill_brewer()+
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
theme(legend.position="none")+
scale_x_continuous(breaks=seq(1950,2030,by=10))Warning: Removed 32000 rows containing missing values (`geom_point()`).
Posterior predictive check with density overlays examines the whole temperature distribution
pp_check(fit_lin, type='dens_overlay', ndraws=20)
LOO-PIT check is good for checking whether the normal distribution is well describing the variation as it is examines the calibration of LOO predictive distributions conditonally on each year. LOO-PIT ploty looks good.
pp_check(fit_lin, type='loo_pit_qq', ndraws=4000)
6 Linear Student’s \(t\) model
The temperatures used in the above analyses are averages over three months, which makes it more likely that they are normally distributed, but there can be extreme events in the feather and we can check whether more robust Student’s \(t\) observation model would give different results (although LOO-PIT check did already indicate that the normal would be good).
fit_lin_t <- brm(temp ~ year, data = data_lin, family = student(),
prior = prior(student_t(3, 0, 0.03), class='b'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics. The b_year posterior looks similar as before and the posterior for degrees of freedom nu has most of the posterior mass for quite large values indicating there is no strong support for thick tailed variation in average summer temperatures.
fit_lin_t Family: student
Links: mu = identity; sigma = identity; nu = identity
Formula: temp ~ year
Data: data_lin (Number of observations: 71)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -34.01 12.27 -58.50 -9.31 1.00 3979 2893
year 0.02 0.01 0.01 0.03 1.00 3979 2923
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 1.03 0.10 0.86 1.24 1.00 3209 2302
nu 24.54 14.36 6.36 60.80 1.00 2972 2325
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).7 Pareto-smoothed importance-sampling leave-one-out cross-validation (PSIS-LOO)
We can use leave-one-out cross-validation to compare the expected predictive performance.
LOO comparison shows normal and Student’s \(t\) model have similar performance.
loo_compare(loo(fit_lin), loo(fit_lin_t)) elpd_diff se_diff
fit_lin 0.0 0.0
fit_lin_t -0.4 0.3 8 Heteroskedastic linear model
Heteroskedasticity assumes that the variation around the linear mean can also vary. We can allow sigma to depend on year, too. Although the additional component is written as sigma ~ year, the log link function is used and the model is for log(sigma). bf() allows listing several formulas.
fit_lin_h <- brm(bf(temp ~ year,
sigma ~ year),
data = data_lin, family = gaussian(),
prior = prior(student_t(3, 0, 0.03), class='b'),
seed = SEED, refresh = 0)Check the summary of the posterior and inference diagnostics. The b_year posterior looks similar as before. The posterior for sigma_year looks like having mosst of the ma for negative values, indicating decrease in temperature variation around the mean.
fit_lin_h Family: gaussian
Links: mu = identity; sigma = log
Formula: temp ~ year
sigma ~ year
Data: data_lin (Number of observations: 71)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -36.37 12.49 -61.25 -10.49 1.00 3412 2842
sigma_Intercept 19.10 8.69 1.56 35.80 1.00 3818 2899
year 0.02 0.01 0.01 0.04 1.00 3426 2885
sigma_year -0.01 0.00 -0.02 -0.00 1.00 3810 2855
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Histogram of b_year and b_sigma_year
as_draws_df(fit_lin_h) |>
mcmc_areas(pars=c('b_year', 'b_sigma_year'))
As log(x) is almost linear when x is close to zero, we can see that the sigma is decreasing about 1% per year (95% interval from 0% to 2%).
Plot posterior predictive distribution at each year until 2030 add_predicted_draws() takes the years from the data and uses fit_lin_h to make the predictions.
data_lin |>
add_row(year=2023:2030) |>
add_predicted_draws(fit_lin_h) |>
# plot data
ggplot(aes(x=year, y=temp)) +
geom_point(color=2) +
# plot lineribbon for the linear model
stat_lineribbon(aes(y = .prediction), .width = c(.95), alpha = 1/2, color=brewer.pal(5, "Blues")[[5]]) +
# decoration
scale_fill_brewer()+
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
theme(legend.position="none")+
scale_x_continuous(breaks=seq(1950,2030,by=10))Warning: Removed 32000 rows containing missing values (`geom_point()`).
Make prior sensitivity analysis by power-scaling both prior and likelihood.
powerscale_sensitivity(fit_lin_h)$sensitivity |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 4 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 b_Intercept 0.03 0.11 -
2 b_sigma_Intercept 0.00 0.10 -
3 b_year 0.03 0.11 -
4 b_sigma_year 0.00 0.11 - We can use leave-one-out cross-validation to compare the expected predictive performance.
LOO comparison shows homoskedastic normal and heteroskedastic normal models have similar performances.
loo_compare(loo(fit_lin), loo(fit_lin_h)) elpd_diff se_diff
fit_lin_h 0.0 0.0
fit_lin -1.6 1.6 9 Heteroskedastic non-linear model
We can test the linearity assumption by using non-linear spline functions, by uing s(year) terms. Sampling is slower as the posterior gets more complex.
fit_spline_h <- brm(bf(temp ~ s(year),
sigma ~ s(year)),
data = data_lin, family = gaussian(),
seed = SEED, refresh = 0)We get warnings about divergences, and try rerunning with higher adapt_delta, which leads to using smaller step sizes. Often adapt_delta=0.999 leads to very slow sampling, but with this small data, this is not an issue.
fit_spline_h <- update(fit_spline_h, control = list(adapt_delta=0.999))Check the summary of the posterior and inference diagnostics. We’re not anymore able to make interpretation of the temperature increase based on this summary. For splines, we see prior scales sds for the spline coefficients.
fit_spline_h Family: gaussian
Links: mu = identity; sigma = log
Formula: temp ~ s(year)
sigma ~ s(year)
Data: data_lin (Number of observations: 71)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Smooth Terms:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sds(syear_1) 1.04 0.95 0.04 3.55 1.00 1943 1987
sds(sigma_syear_1) 0.95 0.91 0.03 3.43 1.00 1365 2147
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 9.42 0.13 9.16 9.68 1.00 5309 2967
sigma_Intercept 0.04 0.09 -0.12 0.21 1.00 4293 2896
syear_1 2.85 2.66 -2.76 8.35 1.00 2337 2216
sigma_syear_1 -1.03 2.37 -6.48 3.83 1.00 1895 1241
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).We can still plot posterior predictive distribution at each year until 2030 add_predicted_draws() takes the years from the data and uses fit_lin_h to make the predictions.
data_lin |>
add_row(year=2023:2030) |>
add_predicted_draws(fit_spline_h) |>
# plot data
ggplot(aes(x=year, y=temp)) +
geom_point(color=2) +
# plot lineribbon for the linear model
stat_lineribbon(aes(y = .prediction), .width = c(.95), alpha = 1/2, color=brewer.pal(5, "Blues")[[5]]) +
# decoration
scale_fill_brewer()+
labs(x= "Year", y = 'Summer temp. @Kilpisjärvi') +
theme(legend.position="none")+
scale_x_continuous(breaks=seq(1950,2030,by=10))Warning: Removed 32000 rows containing missing values (`geom_point()`).
And we can use leave-one-out cross-validation to compare the expected predictive performance.
LOO comparison shows homoskedastic normal linear and heteroskedastic normal spline models have similar performances. There are not enough observations to make clear difference between the models.
loo_compare(loo(fit_lin), loo(fit_spline_h)) elpd_diff se_diff
fit_spline_h 0.0 0.0
fit_lin -0.7 1.8 For spline and other non-parametric models, we can use predictive estimates and predictions to get interpretable quantities. Let’s examine the difference of estimated average temperature in years 1952 and 2022.
temp_diff <- posterior_epred(fit_spline_h, newdata=filter(data_lin,year==1952|year==2022)) |>
rvar() |>
diff() |>
as_draws_df() |>
set_variables('temp_diff')
temp_diff <- data_lin |>
filter(year==1952|year==2022) |>
add_epred_draws(fit_spline_h) |>
pivot_wider(id_cols=.draw, names_from = year, values_from = .epred) |>
mutate(temp_diff = `2022`-`1952`,
.chain = (.draw - 1) %/% 1000 + 1,
.iteration = (.draw - 1) %% 1000 + 1) |>
as_draws_df() |>
subset_draws(variable='temp_diff')Posterior distribution for average summer temperature increase from 1952 to 2022
temp_diff |>
mcmc_hist()
95% posterior interval for average summer temperature increase from 1952 to 2022
temp_diff |>
summarise_draws(~quantile(.x, probs = c(0.025, 0.975)),
~mcse_quantile(.x, probs = c(0.025, 0.975)),
.num_args = list(digits = 2, notation = "dec"))# A tibble: 1 × 5
variable `2.5%` `97.5%` mcse_q2.5 mcse_q97.5
<chr> <dbl> <dbl> <dbl> <dbl>
1 temp_diff 0.51 2.58 0.03 0.03Make prior sensitivity analysis by power-scaling both prior and likelihood with focus on average summer temperature increase from 1952 to 2022.
powerscale_sensitivity(fit_spline_h, prediction = \(x, ...) temp_diff, num_args=list(digits=2)
)$sensitivity |>
filter(variable=='temp_diff') |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 1 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 temp_diff 0.01 0.08 - Probability that the average summer temperature has increased from 1952 to 2022 is 99.5%.
temp_diff |>
mutate(I_temp_diff_gt_0 = temp_diff>0,
temp_diff = NULL) |>
subset_draws(variable='I_temp_diff_gt_0') |>
summarise_draws(mean, mcse_mean)# A tibble: 1 × 3
variable mean mcse_mean
<chr> <dbl> <dbl>
1 I_temp_diff_gt_0 0.997 0.00089510 Comparison of k groups with hierarchical normal models
Load factory data, which contain 5 quality measurements for each of 6 machines. We’re interested in analysing are the quality differences between the machines.
factory <- read.table(url('https://raw.githubusercontent.com/avehtari/BDA_course_Aalto/master/rpackage/data-raw/factory.txt'))
colnames(factory) <- 1:6
factory 1 2 3 4 5 6
1 83 117 101 105 79 57
2 92 109 93 119 97 92
3 92 114 92 116 103 104
4 46 104 86 102 79 77
5 67 87 67 116 92 100We pivot the data to long format
factory <- factory |>
pivot_longer(cols = everything(),
names_to = 'machine',
values_to = 'quality')
factory# A tibble: 30 × 2
machine quality
<chr> <int>
1 1 83
2 2 117
3 3 101
4 4 105
5 5 79
6 6 57
7 1 92
8 2 109
9 3 93
10 4 119
# ℹ 20 more rows10.1 Pooled model
As comparison make also pooled model
fit_pooled <- brm(quality ~ 1, data = factory, refresh=0)Check the summary of the posterior and inference diagnostics.
fit_pooled Family: gaussian
Links: mu = identity; sigma = identity
Formula: quality ~ 1
Data: factory (Number of observations: 30)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 92.88 3.24 86.49 99.17 1.00 2540 2201
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 18.40 2.44 14.40 23.79 1.00 3198 2431
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).10.2 Separate model
As comparison make also seprate model. To make it completely separate we need to have different sigma for each machine, too.
fit_separate <- brm(bf(quality ~ 0 + machine,
sigma ~ 0 + machine),
data = factory, refresh=0)Check the summary of the posterior and inference diagnostics.
fit_separate Family: gaussian
Links: mu = identity; sigma = log
Formula: quality ~ 0 + machine
sigma ~ 0 + machine
Data: factory (Number of observations: 30)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
machine1 76.38 11.18 53.55 100.54 1.00 2371 1909
machine2 106.11 8.85 91.08 121.82 1.00 1381 892
machine3 88.44 10.95 71.84 105.79 1.00 1023 431
machine4 111.57 5.04 101.79 121.61 1.00 2019 1464
machine5 90.13 6.81 77.23 103.60 1.00 2184 1561
machine6 86.14 11.50 62.99 108.31 1.00 2225 1605
sigma_machine1 3.10 0.39 2.44 3.97 1.00 2793 2123
sigma_machine2 2.61 0.41 1.97 3.57 1.00 1558 899
sigma_machine3 2.69 0.42 2.04 3.64 1.00 1408 788
sigma_machine4 2.16 0.41 1.52 3.12 1.00 1976 1301
sigma_machine5 2.50 0.41 1.87 3.43 1.00 2082 1366
sigma_machine6 3.08 0.39 2.44 4.00 1.00 2122 1945
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).11 Common variance hierarchical model (ANOVA)
fit_hier <- brm(quality ~ 1 + (1 | machine),
data = factory, refresh = 0)Check the summary of the posterior and inference diagnostics.
fit_hier Family: gaussian
Links: mu = identity; sigma = identity
Formula: quality ~ 1 + (1 | machine)
Data: factory (Number of observations: 30)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Group-Level Effects:
~machine (Number of levels: 6)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 12.68 5.94 2.96 27.27 1.00 1030 1090
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 92.96 5.71 81.53 104.27 1.00 1263 1326
Family Specific Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 15.09 2.37 11.29 20.68 1.00 1869 2103
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).LOO comparison shows the hierarchical model is the best. The differences are small as the number of observations is small and there is a considerable prediction (aleatoric) uncertainty.
loo_compare(loo(fit_pooled), loo(fit_separate), loo(fit_hier))Warning: Found 3 observations with a pareto_k > 0.7 in model 'fit_separate'. It
is recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations. elpd_diff se_diff
fit_hier 0.0 0.0
fit_separate -3.2 2.6
fit_pooled -3.9 2.0 Different model posterior distributions for the mean quality. Pooled model ignores the varition between machines. Separate model doesn’t take benefit from the similariy of the machines and has higher uncertainty.
ph <- fit_hier |>
spread_rvars(b_Intercept, r_machine[machine,]) |>
mutate(machine_mean = b_Intercept + r_machine) |>
ggplot(aes(xdist=machine_mean, y=machine)) +
stat_halfeye() +
scale_y_continuous(breaks=1:6) +
labs(x='Quality', y='Machine', title='Hierarchical')
ps <- fit_separate |>
as_draws_df() |>
subset_draws(variable='b_machine', regex=TRUE) |>
set_variables(paste0('b_machine[', 1:6, ']')) |>
as_draws_rvars() |>
spread_rvars(b_machine[machine]) |>
mutate(machine_mean = b_machine) |>
ggplot(aes(xdist=machine_mean, y=machine)) +
stat_halfeye() +
scale_y_continuous(breaks=1:6) +
labs(x='Quality', y='Machine', title='Separate')
pp <- fit_pooled |>
spread_rvars(b_Intercept) |>
mutate(machine_mean = b_Intercept) |>
ggplot(aes(xdist=machine_mean, y=0)) +
stat_halfeye() +
scale_y_continuous(breaks=NULL) +
labs(x='Quality', y='All machines', title='Pooled')
(pp / ps / ph) * xlim(c(50,140))Warning: Removed 998 rows containing missing values (`geom_slabinterval()`).
Make prior sensitivity analysis by power-scaling both prior and likelihood with focus on mean quality of each machine. We see no prior sensitivity.
machine_mean <- fit_hier |>
as_draws_df() |>
mutate(across(matches('r_machine'), ~ .x - b_Intercept)) |>
subset_draws(variable='r_machine', regex=TRUE) |>
set_variables(paste0('machine_mean[', 1:6, ']'))
powerscale_sensitivity(fit_hier, prediction = \(x, ...) machine_mean, num_args=list(digits=2)
)$sensitivity |>
filter(str_detect(variable,'machine_mean')) |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 6 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 machine_mean[1] 0.02 0.10 -
2 machine_mean[2] 0.03 0.07 -
3 machine_mean[3] 0.02 0.04 -
4 machine_mean[4] 0.03 0.10 -
5 machine_mean[5] 0.02 0.04 -
6 machine_mean[6] 0.02 0.06 - 12 Hierarchical binomial model
Sorafenib Toxicity Dataset in metadat R package includes results from 13 studies investigating the occurrence of dose limiting toxicities (DLTs) at different doses of Sorafenib.
Load data
load(url('https://github.com/wviechtb/metadat/raw/master/data/dat.ursino2021.rda'))
head(dat.ursino2021) study year dose events total
1 Awada 2005 100 0 4
2 Awada 2005 200 0 3
3 Awada 2005 300 1 5
4 Awada 2005 400 1 10
5 Awada 2005 600 7 12
6 Awada 2005 800 1 3Number of patients per study
dat.ursino2021 |>
group_by(study) |>
summarise(N = sum(total)) |>
ggplot(aes(x=N, y=study)) +
geom_col(fill=4) +
labs(x='Number of patients per study', y='Study')
Distribution of doses
dat.ursino2021 |>
ggplot(aes(x=dose)) +
geom_histogram(breaks=seq(50,1050,by=100), fill=4, colour=1) +
labs(x='Dose (mg)', y='Count') +
scale_x_continuous(breaks=seq(100,1000,by=100))
Each study is using \(2--6\) different dose levels. Three studies that include only two dose levels are likelly to provide weak information on slope.
crosstab <- with(dat.ursino2021,table(dose,study))
data.frame(count=colSums(crosstab), study=colnames(crosstab)) |>
ggplot(aes(x=count, y=study)) +
geom_col(fill=4) +
labs(x='Number of dose levels per study', y='Study')
Pooled model assumes all studies have the same dose effect (reminder: ~ dose is equivalent to ~ 1 + dose)
fit_pooled <- brm(events | trials(total) ~ dose,
prior = c(prior(student_t(7, 0, 1.5), class='Intercept'),
prior(normal(0, 1), class='b')),
family=binomial(), data=dat.ursino2021)Check the summary of the posterior and inference diagnostics.
fit_pooled Family: binomial
Links: mu = logit
Formula: events | trials(total) ~ dose
Data: dat.ursino2021 (Number of observations: 49)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -3.17 0.37 -3.93 -2.46 1.00 1465 1690
dose 0.00 0.00 0.00 0.01 1.00 2921 2417
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Dose coefficient seems to be very small. Looking at the posterior, we see that it is positive with high probability.
fit_pooled |>
as_draws() |>
subset_draws(variable='b_dose') |>
summarise_draws(~quantile(.x, probs = c(0.025, 0.975)), ~mcse_quantile(.x, probs = c(0.025, 0.975)))# A tibble: 1 × 5
variable `2.5%` `97.5%` mcse_q2.5 mcse_q97.5
<chr> <dbl> <dbl> <dbl> <dbl>
1 b_dose 0.00230 0.00520 0.0000262 0.0000351The dose was reported in mg, and most values are in hundreds. It is often sensible to switch to a scale in which the range of values is closer to unit range. In this case it is natural to use g instead of mg.
dat.ursino2021 <- dat.ursino2021 |>
mutate(doseg = dose/1000)Fit the pooled model again uing doseg
fit_pooled <- brm(events | trials(total) ~ doseg,
prior = c(prior(student_t(7, 0, 1.5), class='Intercept'),
prior(normal(0, 1), class='b')),
family=binomial(), data=dat.ursino2021)Check the summary of the posterior and inference diagnostics.
fit_pooled Family: binomial
Links: mu = logit
Formula: events | trials(total) ~ doseg
Data: dat.ursino2021 (Number of observations: 49)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -2.58 0.29 -3.16 -2.03 1.00 2268 2665
doseg 2.41 0.59 1.27 3.55 1.00 2938 2546
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Now it is easier to interpret the presented values. Separate model assumes all studies have different dose effect. It would be a bit complicated to set a different prior on study specific intercepts and other coefficients, so we use the same prior for all.
fit_separate <- brm(events | trials(total) ~ 0 + study + doseg:study,
prior=prior(student_t(7, 0, 1.5), class='b'),
family=binomial(), data=dat.ursino2021)Check the summary of the posterior and inference diagnostics.
fit_separate Family: binomial
Links: mu = logit
Formula: events | trials(total) ~ 0 + study + doseg:study
Data: dat.ursino2021 (Number of observations: 49)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
studyAwada -1.68 0.69 -3.14 -0.43 1.00 4698
studyBorthakurMA -2.26 0.91 -4.17 -0.55 1.00 4941
studyBorthakurMB -1.39 0.82 -3.10 0.14 1.00 4251
studyChen -2.23 1.00 -4.34 -0.39 1.00 5939
studyClark -1.65 0.81 -3.41 -0.18 1.00 4620
studyCrumpMA -2.02 0.75 -3.62 -0.70 1.00 5868
studyCrumpMB -1.57 0.71 -3.07 -0.25 1.00 5238
studyFuruse -2.67 0.96 -4.82 -0.99 1.00 4835
studyMiller -1.03 0.50 -2.02 -0.04 1.00 3747
studyMinami -2.26 0.79 -3.94 -0.83 1.00 5252
studyMoore -1.71 0.72 -3.24 -0.41 1.00 4486
studyNabors -1.80 0.90 -3.79 -0.20 1.00 3954
studyStrumberg -1.73 0.63 -3.06 -0.55 1.00 4579
studyAwada:doseg 1.62 1.30 -0.75 4.36 1.00 4634
studyBorthakurMA:doseg -0.07 1.52 -3.03 3.03 1.00 5063
studyBorthakurMB:doseg 0.10 1.43 -2.68 3.03 1.00 4596
studyChen:doseg -0.77 1.76 -4.63 2.32 1.00 6400
studyClark:doseg 1.55 1.42 -1.04 4.59 1.00 4668
studyCrumpMA:doseg -0.32 1.55 -3.54 2.65 1.00 6677
studyCrumpMB:doseg 0.24 1.36 -2.43 2.97 1.00 5786
studyFuruse:doseg -0.49 1.70 -3.95 2.94 1.00 5489
studyMiller:doseg 0.00 1.45 -3.02 2.84 1.00 4094
studyMinami:doseg -0.28 1.49 -3.38 2.58 1.00 5360
studyMoore:doseg 0.54 1.41 -2.11 3.44 1.00 4827
studyNabors:doseg 1.33 1.26 -0.95 4.09 1.00 4099
studyStrumberg:doseg 0.38 1.13 -1.89 2.74 1.00 4219
Tail_ESS
studyAwada 3206
studyBorthakurMA 3152
studyBorthakurMB 2976
studyChen 3211
studyClark 2805
studyCrumpMA 3120
studyCrumpMB 3322
studyFuruse 2678
studyMiller 2219
studyMinami 2903
studyMoore 3102
studyNabors 2520
studyStrumberg 2744
studyAwada:doseg 2931
studyBorthakurMA:doseg 3251
studyBorthakurMB:doseg 2998
studyChen:doseg 2817
studyClark:doseg 2548
studyCrumpMA:doseg 3187
studyCrumpMB:doseg 3137
studyFuruse:doseg 2676
studyMiller:doseg 2458
studyMinami:doseg 2824
studyMoore:doseg 2958
studyNabors:doseg 2473
studyStrumberg:doseg 2667
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).We build two different hierarchical models. The first one has hierarchical model for the intercept, that is, each study has a parameter telling how much that study differs from the common population intercept.
fit_hier1 <- brm(events | trials(total) ~ doseg + (1 | study),
prior=c(prior(student_t(7, 0, 1.5), class='Intercept'),
prior(normal(0, 1), class='b')),
family=binomial(), data=dat.ursino2021)The second hierarchical model assumes that also the slope can vary between the studies.
fit_hier2 <- brm(events | trials(total) ~ doseg + (doseg | study),
prior=c(prior(student_t(7, 0, 1.5), class='Intercept'),
prior(normal(0, 1), class='b')),
family=binomial(), data=dat.ursino2021)We seem some divergences due to highly varying posterior curvature. We repeat the sampling with higher adapt_delta, which adjust the step size to be smaller. Higher adapt_delta makes the computation slower, but that is not an issue in this case. If you get divergences with adapt_delta=0.99, it is likely that even larger values don’t help, and you need to consider different parameterisation, different model, or more informative priors.
fit_hier2 <- update(fit_hier2, control=list(adapt_delta=0.99))LOO-CV comparison
loo_compare(loo(fit_pooled), loo(fit_separate), loo(fit_hier1), loo(fit_hier2))Warning: Found 6 observations with a pareto_k > 0.7 in model 'fit_separate'. It
is recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations.Warning: Found 1 observations with a pareto_k > 0.7 in model 'fit_hier2'. It is
recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations. elpd_diff se_diff
fit_hier1 0.0 0.0
fit_pooled -0.9 1.7
fit_hier2 -0.9 0.6
fit_separate -15.5 2.9 We get warnings about several Pareto k’s > 0.7 in PSIS-LOO for separate model, but as in that case the LOO-CV estimate is usually overoptimistic and the separate model is the worst, there is no need to use more accurate computation for the separate model.
We get warnings about a few Pareto k’s > 0.7 in PSIS-LOO for both hierarchical models. We can improve the accuracy be running MCMC for these LOO folds. We use add_criterion() function to store the LOO computation results as they take a bit longer now. We get some divergences in case of the second hierarchical model, as leaving out an observation for a study that has only two dose levels is making the posterior having a difficult shape.
fit_hier1 <- add_criterion(fit_hier1, criterion='loo', reloo=TRUE)
fit_hier2 <- add_criterion(fit_hier2, criterion='loo', reloo=TRUE)We repeat the LOO-CV comparison (without separate model). loo() function is useing the reults added to the fit objects.
loo_compare(loo(fit_pooled), loo(fit_hier1), loo(fit_hier2)) elpd_diff se_diff
fit_hier1 0.0 0.0
fit_hier2 -0.8 0.6
fit_pooled -0.9 1.7 The results did not change much. The first hierarchical model is slightly better than other models, but for predictive purposes there is not much difference (there is high aleatoric uncertainty in the predictions). Adding hiearchical model for the slope, decrased the predictive performance and thus it is likely that there is not enough information about the variation in slopes between studies.
Posterior predictive checking showing the observed and predicted number of events. Rootgram uses square root of counts on y-axis for better scaling. Rootogram is useful for count data when the range of counts is small or moderate.
pp_check(fit_pooled, type = "rootogram") +
labs(title='Pooled model')
pp_check(fit_hier1, type = "rootogram") +
labs(title='Hierarchical model')
pp_check(fit_hier2, type = "rootogram") +
labs(title='Hierarchical model')
We see that the hierarchical models have higher probability for future counts that are bigger than maximum observed count and longer predictive distribution tail. This is natural as uncertainty in the variation between tudies increases predictive uncertainty, too, especially as the number of studies is relatively small.
The population level coefficient posterior given pooled model
plot_posterior_pooled <- mcmc_areas(as_draws_df(fit_pooled), regex_pars='b_doseg') +
geom_vline(xintercept=0, linetype='dashed') +
labs(title='Pooled model')The population level coefficient posterior given hierarchical model 1
plot_posterior_hier1 <- mcmc_areas(as_draws_df(fit_hier1), regex_pars='b_doseg') +
geom_vline(xintercept=0, linetype='dashed') +
labs(title='Hierarchical model 1')The population level coefficient posterior given hierarchical model 3
plot_posterior_hier2 <- mcmc_areas(as_draws_df(fit_hier2), regex_pars='b_doseg') +
geom_vline(xintercept=0, linetype='dashed') +
labs(title='Hierarchical model 2')
(plot_posterior_pooled / plot_posterior_hier1 / plot_posterior_hier2) * xlim(c(0,8.5))Warning: Removed 1 rows containing missing values (`geom_segment()`).
All models agree that the slope is very likely positive. The hierarchical models have more uncertainty, but also higher posterior mean.
When we look at the study specific parameters, we see that the Miller study has slightly higher intercept (leading to higher theta).
(mcmc_areas(as_draws_df(fit_hier1), regex_pars='r_study\\[.*Intercept') +
labs(title='Hierarchical model 1')) /
(mcmc_areas(as_draws_df(fit_hier2), regex_pars='r_study\\[.*Intercept') +
labs(title='Hierarchical model 2'))
There are no clear differences in slopes.
mcmc_areas(as_draws_df(fit_hier2), regex_pars='r_study\\[.*doseg') +
labs(title='Hierarchical model 2')
Based on LOO comparison we could continue with any of the models, but if we want to take into account the unknown possible study variations, it is best to continue with the hierarchical model 2. We could reduce the uncertainty by spending some effort to elicit a more informative priors for the between study variation, by searching open study databses for similar studies. In this example, we skip that and continue with other parts of the workflow.
fit_hier2 |>
powerscale_sequence() |>
powerscale_plot_dens(variables='b_doseg') +
# switch rows and cols
facet_grid(rows=vars(.data$variable),
cols=vars(.data$component)) +
# cleaning
ggtitle(NULL,NULL) +
labs(x='Dose (g) coefficient', y=NULL) +
scale_y_continuous(breaks=NULL) +
theme(axis.line.y=element_blank(),
strip.text.y=element_blank())
Summarise the prior and likelihood sensitivity using cumulative Jensen-Shannon distance focusing on the common population level intercept.
powerscale_sensitivity(fit_hier2, variable='b_doseg'
)$sensitivity |>
mutate(across(where(is.double), ~num(.x, digits=2)))# A tibble: 1 × 4
variable prior likelihood diagnosis
<chr> <num:.2!> <num:.2!> <chr>
1 b_doseg 0.25 0.17 prior-data conflictThe posterior for the probability of event given certain dose and a new study for hierarchical model 2.
data.frame(study='new',
doseg=seq(0.1,1,by=0.1),
total=1) |>
add_linpred_draws(fit_hier2, transform=TRUE, allow_new_levels=TRUE) |>
ggplot(aes(x=doseg, y=.linpred)) +
stat_lineribbon(.width = c(.95), alpha = 1/2, color=brewer.pal(5, "Blues")[[5]]) +
scale_fill_brewer()+
labs(x= "Dose (g)", y = 'Probability of event', title='Hierarchical model') +
theme(legend.position="none") +
geom_hline(yintercept=0) +
scale_x_continuous(breaks=seq(0.1,1,by=0.1)) +
ylim(c(0,0.15))Warning: Removed 24617 rows containing missing values (`stat_slabinterval()`).
If we plot individual posterior draws, we see that there is a lot of uncertainty about the overall probability (explained by the variation in Intercept in different studies), but less uncertainty about the slope.
data.frame(study='new',
doseg=seq(0.1,1,by=0.1),
total=1) |>
add_linpred_draws(fit_hier2, transform=TRUE, allow_new_levels=TRUE, ndraws=100) |>
ggplot(aes(x=doseg, y=.linpred)) +
geom_line(aes(group=.draw), alpha = 1/2, color = brewer.pal(5, "Blues")[[3]])+
scale_fill_brewer()+
labs(x= "Dose (g)", y = 'Probability of event') +
theme(legend.position="none") +
geom_hline(yintercept=0) +
scale_x_continuous(breaks=seq(0.1,1,by=0.1))
13 Hierarchical binomial model 2
Studies on Pharmacologic Treatments for Chronic Obstructive Pulmonary Disease includes results from 39 trials examining pharmacologic treatments for chronic obstructive pulmonary disease (COPD).
Load data
load(url('https://github.com/wviechtb/metadat/raw/master/data/dat.baker2009.rda'))
# force character strings to factors for easier ploting
dat.baker2009 <- dat.baker2009 |>
mutate(study = factor(study),
treatment = factor(treatment),
id = factor(id))Look at six first lines of the data frame
head(dat.baker2009) study year id treatment exac total
1 Llewellyn-Jones 1996 1996 1 Fluticasone 0 8
2 Llewellyn-Jones 1996 1996 1 Placebo 3 8
3 Boyd 1997 1997 2 Salmeterol 47 229
4 Boyd 1997 1997 2 Placebo 59 227
5 Paggiaro 1998 1998 3 Fluticasone 45 142
6 Paggiaro 1998 1998 3 Placebo 51 139Total number of patients in each study varies a lot
dat.baker2009 |>
group_by(study) |>
summarise(N = sum(total)) |>
ggplot(aes(x=N, y=study)) +
geom_col(fill=4) +
labs(x='Number of patients per study', y='Study')
None of the treatments is included in every study, and each study includes \(2--4\) treatments.
crosstab <- with(dat.baker2009,table(study, treatment))
#
plot_treatments <- data.frame(number_of_studies=colSums(crosstab), treatment=colnames(crosstab)) |>
ggplot(aes(x=number_of_studies,y=treatment)) +
geom_col(fill=4) +
labs(x='Number of studies with a treatment X', y='Treatment') +
geom_vline(xintercept=nrow(crosstab), linetype='dashed') +
scale_x_continuous(breaks=c(0,10,20,30,39))
#
plot_studies <- data.frame(number_of_treatments=rowSums(crosstab), study=rownames(crosstab)) |>
ggplot(aes(x=number_of_treatments,y=study)) +
geom_col(fill=4) +
labs(x='Number of treatments in a study Y', y='Study') +
geom_vline(xintercept=ncol(crosstab), linetype='dashed') +
scale_x_continuous(breaks=c(0,2,4,6,8))
#
plot_treatments + plot_studies
The first model is pooling the information over studies, but estimating separate theta for each treatment (including placebo).
fit_pooled <- brm(exac | trials(total) ~ 0 + treatment,
prior = prior(student_t(7, 0, 1.5), class='b'),
family=binomial(), data=dat.baker2009)Check the summary of the posterior and inference diagnostics.
fit_pooled Family: binomial
Links: mu = logit
Formula: exac | trials(total) ~ 0 + treatment
Data: dat.baker2009 (Number of observations: 94)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat
treatmentBudesonide -0.30 0.10 -0.50 -0.12 1.00
treatmentBudesonidePFormoterol -0.49 0.10 -0.68 -0.31 1.00
treatmentFluticasone 0.35 0.04 0.28 0.43 1.00
treatmentFluticasonePSalmeterol 0.12 0.03 0.06 0.18 1.00
treatmentFormoterol -0.71 0.06 -0.83 -0.59 1.00
treatmentPlacebo -0.28 0.02 -0.32 -0.24 1.00
treatmentSalmeterol -0.38 0.03 -0.44 -0.33 1.00
treatmentTiotropium -0.90 0.03 -0.96 -0.84 1.00
Bulk_ESS Tail_ESS
treatmentBudesonide 6266 2832
treatmentBudesonidePFormoterol 6879 3057
treatmentFluticasone 7007 3221
treatmentFluticasonePSalmeterol 7283 2962
treatmentFormoterol 6827 2633
treatmentPlacebo 7126 2947
treatmentSalmeterol 7786 3088
treatmentTiotropium 6762 3076
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Treatment effect posteriors
fit_pooled |>
as_draws_df() |>
subset_draws(variable='b_', regex=TRUE) |>
set_variables(paste0('b_treatment[', levels(factor(dat.baker2009$treatment)), ']')) |>
as_draws_rvars() |>
spread_rvars(b_treatment[treatment]) |>
mutate(theta_treatment = rfun(plogis)(b_treatment)) |>
ggplot(aes(xdist=theta_treatment, y=treatment)) +
stat_halfeye() +
labs(x='theta', y='Treatment', title='Pooled over studies, separate over treatments') 
Treatment effect odds-ratio posteriors
theta <- fit_pooled |>
as_draws_df() |>
subset_draws(variable='b_', regex=TRUE) |>
set_variables(paste0('b_treatment[', levels(factor(dat.baker2009$treatment)), ']')) |>
as_draws_rvars() |>
spread_rvars(b_treatment[treatment]) |>
mutate(theta_treatment = rfun(plogis)(b_treatment))
theta_placebo <- filter(theta,treatment=='Placebo')$theta_treatment[[1]]
theta |>
mutate(treatment_oddsratio = (theta_treatment/(1-theta_treatment))/(theta_placebo/(1-theta_placebo))) |>
filter(treatment != "Placebo") |>
ggplot(aes(xdist=treatment_oddsratio, y=treatment)) +
stat_halfeye() +
labs(x='Odds-ratio', y='Treatment', title='Pooled over studies, separate over treatments') +
geom_vline(xintercept=1, linetype='dashed')
We see a big variation between treatments and two treatments seem to be harmful, which is suspicious. Looking at the data we see that not all studies included all treatments, and thus if some of the studies had more events, then the above estimates can be wrong.
The target is discrete count, but as the range of counts is big, a rootogram would look messy, and density overlay plot is a better choice. Posterior predictive checking with kernel density estimates for the data and 10 posterior predictive replicates shows clear discrepancy.
pp_check(fit_pooled, type='dens_overlay')
Posterior predictive checking with PIT values and ECDF difference plot with envelope shows clear discrepancy.
pp_check(fit_pooled, type='pit_ecdf', ndraws=4000)
Posterior predictive checking with LOO-PIT values show clear discrepancy.
pp_check(fit_pooled, type='loo_pit_qq', ndraws=4000) +
geom_abline() +
ylim(c(0,1))Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.Warning: Removed 9 rows containing missing values (`geom_point()`).Warning: Removed 2 rows containing missing values (`geom_path()`).
The second model uses a hiearchical model both for treatment effects and study effects.
fit_hier <- brm(exac | trials(total) ~ (1 | treatment) + (1 | study),
family=binomial(), data=dat.baker2009)Check the summary of the posterior and inference diagnostics.
fit_hier Family: binomial
Links: mu = logit
Formula: exac | trials(total) ~ (1 | treatment) + (1 | study)
Data: dat.baker2009 (Number of observations: 94)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Group-Level Effects:
~study (Number of levels: 39)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 1.20 0.17 0.93 1.56 1.02 493 1134
~treatment (Number of levels: 8)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept) 0.18 0.07 0.09 0.35 1.00 1172 1880
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept -0.91 0.21 -1.34 -0.50 1.00 412 747
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).LOO-CV comparison
loo_compare(loo(fit_pooled), loo(fit_hier))Warning: Found 22 observations with a pareto_k > 0.7 in model 'fit_pooled'. It
is recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations.Warning: Found 24 observations with a pareto_k > 0.7 in model 'fit_hier'. It is
recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations. elpd_diff se_diff
fit_hier 0.0 0.0
fit_pooled -1948.2 302.5We get warnings about Pareto k’s > 0.7 in PSIS-LOO, but as the difference between the models is huge, we can be confident that the order would the same if we fixed the computation, and the hierarchical model is much better and there is high variation between studies. Clearly there are many highly influential observations.
Posterior predictive checking with kernel density estimates for the data and 10 posterior predictive replicates looks good (although with this many parameters, this check is likely to be optimistic).
pp_check(fit_hier, type='dens_overlay')
Posterior predictive checking with PIT values and ECDF difference plot with envelope looks good (although with this many parameters, this check is likely to be optimistic).
pp_check(fit_hier, type='pit_ecdf', ndraws=4000)
Posterior predictive checking with LOO-PIT values look good (alhough as there are Pareto-khat warnings, it is possible that this diagnostic is optimistic).
pp_check(fit_hier, type='loo_pit_qq', ndraws=4000) +
geom_abline() +
ylim(c(0,1))Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.Warning: Removed 3 rows containing missing values (`geom_point()`).Warning: Removed 2 rows containing missing values (`geom_path()`).
Treatment effect posteriors have now much less variation.
fit_hier |>
spread_rvars(b_Intercept, r_treatment[treatment,]) |>
mutate(theta_treatment = rfun(plogis)(b_Intercept + r_treatment)) |>
ggplot(aes(xdist=theta_treatment, y=treatment)) +
stat_halfeye() +
labs(x='theta', y='Treatment', title='Hierarchical over studies, hierarchical over treatments') 
Study effect posteriors show the expected high variation.
fit_hier |>
spread_rvars(b_Intercept, r_study[study,]) |>
mutate(theta_study = rfun(plogis)(b_Intercept + r_study)) |>
ggplot(aes(xdist=theta_study, y=study)) +
stat_halfeye() +
labs(x='theta', y='Study', title='Hierarchical over studies, hierarchical over treatments') 
Treatment effect odds-ratio posteriors
theta <- fit_hier |>
spread_rvars(b_Intercept, r_treatment[treatment,]) |>
mutate(theta_treatment = rfun(plogis)(b_Intercept + r_treatment))
theta_placebo <- filter(theta,treatment=='Placebo')$theta_treatment[[1]]
theta |>
mutate(treatment_oddsratio = (theta_treatment/(1-theta_treatment))/(theta_placebo/(1-theta_placebo))) |>
filter(treatment != "Placebo") |>
ggplot(aes(xdist=treatment_oddsratio, y=treatment)) +
stat_halfeye() +
labs(x='Odds-ratio', y='Treatment', title='Hierarchical over studies, hierarchical over treatments') +
geom_vline(xintercept=1, linetype='dashed')
Treatment effect odds-ratios look now more reasonable. As now all treatments were compared to placebo, there is less overlap in the distributions as when looking at the thetas, as all thetas include similar uncertainty about the overall theta due to high variation between studies. The third model includes interaction so that the treatment can depend on study.
fit_hier2 <- brm(exac | trials(total) ~ (1 | treatment) + (treatment | study),
family=binomial(), data=dat.baker2009, control=list(adapt_delta=0.9))LOO comparison shows
loo_compare(loo(fit_hier), loo(fit_hier2))Warning: Found 24 observations with a pareto_k > 0.7 in model 'fit_hier'. It is
recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations.Warning: Found 40 observations with a pareto_k > 0.7 in model 'fit_hier2'. It
is recommended to set 'moment_match = TRUE' in order to perform moment matching
for problematic observations. elpd_diff se_diff
fit_hier2 0.0 0.0
fit_hier -3.6 3.4 We get warnings about Pareto k’s > 0.7 in PSIS-LOO, but as the models are similar, and the difference is small, we can be relatively confident that the more complex model is not better.
Licenses
- Code © 2017-2024, Aki Vehtari, licensed under BSD-3.
- Text © 2017-2024, Aki Vehtari, licensed under CC-BY-NC 4.0.
LS0tCnRpdGxlOiAiQmF5ZXNpYW4gZGF0YSBhbmFseXNpcyAtIEJSTVMgZGVtb3MiCmF1dGhvcjogIkFraSBWZWh0YXJpIgpkYXRlOiAiRmlyc3QgdmVyc2lvbiAyMDIzLTEyLTA1LiBMYXN0IG1vZGlmaWVkIGByIGZvcm1hdChTeXMuRGF0ZSgpKWAuIgpvdXRwdXQ6CiAgaHRtbF9kb2N1bWVudDoKICAgIGZpZ19jYXB0aW9uOiB5ZXMKICAgIHRvYzogVFJVRQogICAgdG9jX2RlcHRoOiAyCiAgICBudW1iZXJfc2VjdGlvbnM6IFRSVUUKICAgIHRvY19mbG9hdDoKICAgICAgc21vb3RoX3Njcm9sbDogRkFMU0UKICAgIHRoZW1lOiByZWFkYWJsZQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KIyBTZXR1cCAgey51bm51bWJlcmVkfQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChjYWNoZT1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgZXJyb3I9RkFMU0UsIHdhcm5pbmc9VFJVRSwgY29tbWVudD1OQSwgb3V0LndpZHRoPSc5NSUnKQpgYGAKCioqTG9hZCBwYWNrYWdlcyoqCgpgYGB7cn0KbGlicmFyeSh0aWR5cikKbGlicmFyeShkcGx5cikKbGlicmFyeSh0aWJibGUpCmxpYnJhcnkocGlsbGFyKQpsaWJyYXJ5KHN0cmluZ3IpCmxpYnJhcnkoYnJtcykKb3B0aW9ucyhicm1zLmJhY2tlbmQgPSAiY21kc3RhbnIiLCBtYy5jb3JlcyA9IDIpCmxpYnJhcnkocG9zdGVyaW9yKQpvcHRpb25zKHBpbGxhci5uZWdhdGl2ZSA9IEZBTFNFKQpsaWJyYXJ5KGxvbykKbGlicmFyeShwcmlvcnNlbnNlKQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoYmF5ZXNwbG90KQp0aGVtZV9zZXQoYmF5ZXNwbG90Ojp0aGVtZV9kZWZhdWx0KGJhc2VfZmFtaWx5ID0gInNhbnMiKSkKbGlicmFyeSh0aWR5YmF5ZXMpCmxpYnJhcnkoZ2dkaXN0KQpsaWJyYXJ5KHBhdGNod29yaykKbGlicmFyeShSQ29sb3JCcmV3ZXIpClNFRUQgPC0gNDg5MjcgIyBzZXQgcmFuZG9tIHNlZWQgZm9yIHJlcHJvZHVjYWJpbGl0eQpgYGAKCiMgSW50cm9kdWN0aW9uCgpUaGlzIG5vdGVib29rIGNvbnRhaW5zIHNldmVyYWwgZXhhbXBsZXMgb2YgaG93IHRvIHVzZSBbU3Rhbl0oaHR0cHM6Ly9tYy1zdGFuLm9yZykgaW4gUiB3aXRoIFtfX2JybXNfX10oaHR0cHM6Ly9wYXVsLWJ1ZXJrbmVyLmdpdGh1Yi5pby9icm1zLykuIFRoaXMgbm90ZWJvb2sgYXNzdW1lcyBiYXNpYyBrbm93bGVkZ2Ugb2YgQmF5ZXNpYW4gaW5mZXJlbmNlIGFuZCBNQ01DLiBUaGUgZXhhbXBsZXMgYXJlIHJlbGF0ZWQgdG8gW0JheWVzaWFuIGRhdGEgYW5hbHlzaXMgY291cnNlXShodHRwczovL2F2ZWh0YXJpLmdpdGh1Yi5pby9CREFfY291cnNlX0FhbHRvLykuCgojIEJlcm5vdWxsaSBtb2RlbAoKVG95IGRhdGEgd2l0aCBzZXF1ZW5jZSBvZiBmYWlsdXJlcyAoMCkgYW5kIHN1Y2Nlc3NlcyAoMSkuIFdlIHdvdWxkCmxpa2UgdG8gbGVhcm4gYWJvdXQgdGhlIHVua25vd24gcHJvYmFiaWxpdHkgb2Ygc3VjY2Vzcy4KCmBgYHtyfQpkYXRhX2Jlcm4gPC0gZGF0YS5mcmFtZSh5ID0gYygxLCAxLCAxLCAwLCAxLCAxLCAxLCAwLCAxLCAwKSkKYGBgCgpBcyB1c3VhbCBpbiBjYXNlIG9mIGdlbmVyYWxpemQgbGluZWFyIG1vZGVscywgKEdMTXMpIGJybXMgZGVmaW5lcwp0aGUgcHJpb3JzIG9uIHRoZSBsYXRlbnQgbW9kZWwgcGFyYW1ldGVycy4gV2l0aCBCZXJub3VsbGkgdGhlCmRlZmF1bHQgbGluayBmdW5jdGlvbiBpcyBsb2dpdCwgYW5kIHRodXMgdGhlIHByaW9yIGlzIHNldCBvbgpsb2dpdCh0aGV0YSkuIEFzIHRoZXJlIGFyZSBubyBjb3ZhcmlhdGVzIGxvZ2l0KHRoZXRhKT1JbnRlcmNlcHQuClRoZSBicm1zIGRlZmF1bHQgcHJpb3IgZm9yIEludGVyY2VwdCBpcyBzdHVkZW50X3QoMywgMCwgMi41KSwgYnV0CndlIHVzZSBzdHVkZW50X3QoNywgMCwgMS41KSB3aGljaCBpcyBjbG9zZSB0byBsb2dpc3RpYwpkaXN0cmlidXRpb24sIGFuZCB0aHVzIG1ha2VzIHRoZSBwcmlvciBuZWFyLXVuaWZvcm0gZm9yIHRoZXRhLgpXZSBjYW4gc2ltdWxhdGUgZnJvbSB0aGVzZSBwcmlvcnMgdG8gY2hlY2sgdGhlIGltcGxpZWQgcHJpb3Igb24gdGhldGEuCldlIG5leHQgY29tcGFyZSB0aGUgcmVzdWx0IHRvIHVzaW5nIG5vcm1hbCgwLCAxKSBwcmlvciBvbiBsb2dpdApwcm9iYWJpbGl0eS4gV2UgdmlzdWFsaXplIHRoZSBpbXBsaWVkIHByaW9ycyBieSBzYW1wbGluZyBmcm9tIHRoZSBwcmlvcnMuCgpgYGB7cn0KZGF0YS5mcmFtZSh0aGV0YSA9IHBsb2dpcyhnZ2Rpc3Q6OnJzdHVkZW50X3Qobj0yMDAwMCwgZGY9MywgbXU9MCwgc2lnbWE9Mi41KSkpIHw+CiAgbWNtY19oaXN0KCkgKwogIHhsaW0oYygwLDEpKSArCiAgbGFicyh0aXRsZT0nRGVmYXVsdCBicm1zIHN0dWRlbnRfdCgzLCAwLCAyLjUpIHByaW9yIG9uIEludGVyY2VwdCcpCmRhdGEuZnJhbWUodGhldGEgPSBwbG9naXMoZ2dkaXN0Ojpyc3R1ZGVudF90KG49MjAwMDAsIGRmPTcsIG11PTAsIHNpZ21hPTEuNSkpKSB8PgogIG1jbWNfaGlzdCgpICsKICB4bGltKGMoMCwxKSkgKwogIGxhYnModGl0bGU9J3N0dWRlbnRfdCg3LCAwLCAxLjUpIHByaW9yIG9uIEludGVyY2VwdCcpCmBgYAoKQWxtb3N0IHVuaWZvcm0gcHJpb3Igb24gdGhldGEgY291bGQgYmUgb2J0YWluZWQgYWxzbyB3aXRoIG5vcm1hbCgwLDEuNSkKCmBgYHtyfQpkYXRhLmZyYW1lKHRoZXRhID0gcGxvZ2lzKHJub3JtKG49MjAwMDAsIG1lYW49MCwgc2Q9MS41KSkpIHw+CiAgbWNtY19oaXN0KCkgKwogIHhsaW0oYygwLDEpKSArCiAgbGFicyh0aXRsZT0nbm9ybWFsKDAsIDEuNSkgcHJpb3Igb24gSW50ZXJjZXB0JykKYGBgCgpGb3JtdWxhIGB5IH4gMWAgY29ycmVzcG9uZHMgdG8gYSBtb2RlbCAkXG1hdGhybXtsb2dpdH0oXHRoZXRhKSA9CgpgYGB7cn0KI1xhbHBoYVx0aW1lcyAxID0gXGFscGhhJC4gYGJybXM/IGRlbm90ZXMgdGhlICRcYWxwaGEkIGFzIGBJbnRlcmNlcHRgLgpgYGAKYGBge3IgcmVzdWx0cz0naGlkZSd9CmZpdF9iZXJuIDwtIGJybSh5IH4gMSwgZmFtaWx5ID0gYmVybm91bGxpKCksIGRhdGEgPSBkYXRhX2Jlcm4sCiAgICAgICAgICAgICAgICBwcmlvciA9IHByaW9yKHN0dWRlbnRfdCg3LCAwLCAxLjUpLCBjbGFzcz0nSW50ZXJjZXB0JyksCiAgICAgICAgICAgICAgICBzZWVkID0gU0VFRCwgcmVmcmVzaCA9IDApCmBgYAoKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLgoKYGBge3J9CmZpdF9iZXJuCmBgYAoKRXh0cmFjdCB0aGUgcG9zdGVyaW9yIGRyYXdzCgpgYGB7cn0KZHJhd3MgPC0gYXNfZHJhd3NfZGYoZml0X2Jlcm4pCmBgYAoKV2UgY2FuIGdldCBzdW1tYXJ5IGluZm9ybWF0aW9uIHVzaW5nIHN1bW1hcmlzZV9kcmF3cygpCgpgYGB7cn0KZHJhd3MgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J2JfSW50ZXJjZXB0JykgfD4KICBzdW1tYXJpc2VfZHJhd3MoKQpgYGAKCldlIGNhbiBjb21wdXRlIHRoZSBwcm9iYWJpbGl0eSBvZiBzdWNjZXNzIGJ5IHVzaW5nIHBsb2dpcyB3aGljaCBpcwplcXVhbCB0byBpbnZlcnNlLWxvZ2l0IGZ1bmN0aW9uCgpgYGB7cn0KZHJhd3MgPC0gZHJhd3MgfD4KICBtdXRhdGVfdmFyaWFibGVzKHRoZXRhPXBsb2dpcyhiX0ludGVyY2VwdCkpCmBgYAoKU3VtbWFyeSBvZiB0aGV0YSBieSB1c2luZyBzdW1tYXJpc2VfZHJhd3MoKQoKYGBge3J9CmRyYXdzIHw+CiAgc3Vic2V0X2RyYXdzKHZhcmlhYmxlPSd0aGV0YScpIHw+CiAgc3VtbWFyaXNlX2RyYXdzKCkKYGBgCgpIaXN0b2dyYW0gb2YgdGhldGEKCmBgYHtyfQptY21jX2hpc3QoZHJhd3MsIHBhcnM9J3RoZXRhJykgKwogIHhsYWIoJ3RoZXRhJykgKwogIHhsaW0oYygwLDEpKQpgYGAKClByaW9yIGFuZCBsaWtlbGlob29kIHNlbnNpdGl2aXR5IHBsb3Qgc2hvd3MgcG9zdGVyaW9yIGRlbnNpdHkgZXN0aW1hdGUKZGVwZW5kaW5nIG9uIGFtb3VudCBvZiBwb3dlci1zY2FsaW5nLiBPdmVybGFwcGluZyBsaW5lIGluZGljYXRlIGxvdwpzZW5zaXRpdml0eSBhbmQgd2lkZXIgZ2FwcyBiZXR3ZWVuIGxpbmUgaW5kaWNhdGUgZ3JlYXRlciBzZW5zaXRpdml0eS4KCmBgYHtyfQp0aGV0YSA8LSBkcmF3cyB8PgogIHN1YnNldF9kcmF3cyh2YXJpYWJsZT0ndGhldGEnKQpwb3dlcnNjYWxlX3NlcXVlbmNlKGZpdF9iZXJuLCBwcmVkaWN0aW9uID0gXCh4LCAuLi4pIHRoZXRhKSB8PgogIHBvd2Vyc2NhbGVfcGxvdF9kZW5zKHZhcmlhYmxlcz0ndGhldGEnKSArCiAgIyBzd2l0Y2ggcm93cyBhbmQgY29scwogIGZhY2V0X2dyaWQocm93cz12YXJzKC5kYXRhJHZhcmlhYmxlKSwKICAgICAgICAgICAgIGNvbHM9dmFycyguZGF0YSRjb21wb25lbnQpKSArCiAgIyBjbGVhbmluZwogIGdndGl0bGUoTlVMTCxOVUxMKSArCiAgbGFicyh4PSd0aGV0YScsIHk9TlVMTCkgKwogIHNjYWxlX3lfY29udGludW91cyhicmVha3M9TlVMTCkgKwogIHRoZW1lKGF4aXMubGluZS55PWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBzdHJpcC50ZXh0Lnk9ZWxlbWVudF9ibGFuaygpKSArCiAgeGxpbShjKDAsMSkpCmBgYAoKV2UgY2FuIHN1bW1hcmlzZSB0aGUgcHJpb3IgYW5kIGxpa2VsaWhvb2Qgc2Vuc2l0aXZpdHkgdXNpbmcKY3VtdWxhdGl2ZSBKZW5zZW4tU2hhbm5vbiBkaXN0YW5jZS4KCmBgYHtyfQpwb3dlcnNjYWxlX3NlbnNpdGl2aXR5KGZpdF9iZXJuLCBwcmVkaWN0aW9uID0gXCh4LCAuLi4pIHRoZXRhKSRzZW5zaXRpdml0eSB8PgogICAgICAgICAgICAgICAgICAgICAgICAgZmlsdGVyKHZhcmlhYmxlPT0ndGhldGEnKSB8PgogICAgICAgICAgICAgICAgICAgICAgICAgbXV0YXRlKGFjcm9zcyh3aGVyZShpcy5kb3VibGUpLCAgfm51bSgueCwgZGlnaXRzPTIpKSkKYGBgCgoKIyBCaW5vbWlhbCBtb2RlbAoKSW5zdGVhZCBvZiBzZXF1ZW5jZSBvZiAwJ3MgYW5kIDEncywgd2UgY2FuIHN1bW1hcml6ZSB0aGUgZGF0YSB3aXRoCnRoZSBudW1iZXIgb2YgdHJpYWxzIGFuZCB0aGUgbnVtYmVyIHN1Y2Nlc3NlcyBhbmQgdXNlIEJpbm9taWFsCm1vZGVsLiBUaGUgcHJpb3IgaXMgc3BlY2lmaWVkIGluIHRoZSAnbGF0ZW50IHNwYWNlJy4gVGhlIGFjdHVhbApwcm9iYWJpbGl0eSBvZiBzdWNjZXNzLCB0aGV0YSA9IHBsb2dpcyhhbHBoYSksIHdoZXJlIHBsb2dpcyBpcyB0aGUKaW52ZXJzZSBvZiB0aGUgbG9naXN0aWMgZnVuY3Rpb24uCgpCaW5vbWlhbCBtb2RlbCB3aXRoIHRoZSBzYW1lIGRhdGEgYW5kIHByaW9yCgpgYGB7cn0KZGF0YV9iaW4gPC0gZGF0YS5mcmFtZShOID0gYygxMCksIHkgPSBjKDcpKQpgYGAKCkZvcm11bGEgYHkgfCB0cmlhbHMoTikgfiAxYCBjb3JyZXNwb25kcyB0byBhIG1vZGVsCiRcbWF0aHJte2xvZ2l0fShcdGhldGEpID0gXGFscGhhJCwgYW5kIHRoZSBudW1iZXIgb2YgdHJpYWxzIGZvcgplYWNoIG9ic2VydmF0aW9uIGlzIHByb3ZpZGVkIGJ5IGB8IHRyaWFscyhOKWAKCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfYmluIDwtIGJybSh5IHwgdHJpYWxzKE4pIH4gMSwgZmFtaWx5ID0gYmlub21pYWwoKSwgZGF0YSA9IGRhdGFfYmluLAogICAgICAgICAgICAgICBwcmlvciA9IHByaW9yKHN0dWRlbnRfdCg3LCAwLDEuNSksIGNsYXNzPSdJbnRlcmNlcHQnKSwKICAgICAgICAgICAgICAgc2VlZCA9IFNFRUQsIHJlZnJlc2ggPSAwKQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4KCmBgYHtyfQpmaXRfYmluCmBgYAoKVGhlIGRpYWdub3N0aWMgaW5kaWNhdGVzIHByaW9yLWRhdGEgY29uZmxpY3QsIHRoYXQgaXMsIGJvdGggcHJpb3IKYW5kIGxpa2VsaWhvb2QgYXJlIGluZm9ybWF0aXZlLiBJZiB0aGVyZSBpcyB0cnVlIHN0cm9uZyBwcmlvcgppbmZvcm1hdGlvbiB0aGF0IHdvdWxkIGp1c3RpZnkgdGhlIG5vcm1hbCgwLDEpIHByaW9yLCB0aGVuIHRoaXMgaXMKZmluZSwgYnV0IG90aGVyd2lzZSBtb3JlIHRoaW5raW5nIGlzIHJlcXVpcmVkIChnb2FsIGlzIG5vdCBhZGp1c3QKcHJpb3IgdG8gcmVtb3ZlIGRpYWdub3N0aWMgd2FybmluZ3Mgd2l0aG95dCB0aGlua2luZykuIEluIHRoaXMgdG95CmV4YW1wbGUsIHdlIHByb2NlZWQgd2l0aCB0aGlzIHByaW9yLgoKRXh0cmFjdCB0aGUgcG9zdGVyaW9yIGRyYXdzCgpgYGB7cn0KZHJhd3MgPC0gYXNfZHJhd3NfZGYoZml0X2JpbikKYGBgCgpXZSBjYW4gZ2V0IHN1bW1hcnkgaW5mb3JtYXRpb24gdXNpbmcgc3VtbWFyaXNlX2RyYXdzKCkKCmBgYHtyfQpkcmF3cyB8PgogIHN1YnNldF9kcmF3cyh2YXJpYWJsZT0nYl9JbnRlcmNlcHQnKSB8PgogIHN1bW1hcmlzZV9kcmF3cygpCmBgYAoKV2UgY2FuIGNvbXB1dGUgdGhlIHByb2JhYmlsaXR5IG9mIHN1Y2Nlc3MgYnkgdXNpbmcgcGxvZ2lzIHdoaWNoIGlzCmVxdWFsIHRvIGludmVyc2UtbG9naXQgZnVuY3Rpb24KCmBgYHtyfQpkcmF3cyA8LSBkcmF3cyB8PgogIG11dGF0ZV92YXJpYWJsZXModGhldGE9cGxvZ2lzKGJfSW50ZXJjZXB0KSkKYGBgCgpTdW1tYXJ5IG9mIHRoZXRhIGJ5IHVzaW5nIHN1bW1hcmlzZV9kcmF3cygpCgpgYGB7cn0KZHJhd3MgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J3RoZXRhJykgfD4KICBzdW1tYXJpc2VfZHJhd3MoKQpgYGAKCkhpc3RvZ3JhbSBvZiB0aGV0YQoKYGBge3J9Cm1jbWNfaGlzdChkcmF3cywgcGFycz0ndGhldGEnKSArCiAgeGxhYigndGhldGEnKSArCiAgeGxpbShjKDAsMSkpCmBgYAoKUmUtcnVuIHRoZSBtb2RlbCB3aXRoIGEgbmV3IGRhdGEgZGF0YXNldCB3aXRob3V0IHJlY29tcGlsaW5nCgpgYGB7cn0KZGF0YV9iaW4gPC0gZGF0YS5mcmFtZShOID0gYyg1KSwgeSA9IGMoNCkpCmZpdF9iaW4gPC0gdXBkYXRlKGZpdF9iaW4sIG5ld2RhdGEgPSBkYXRhX2JpbikKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuCgpgYGB7cn0KZml0X2JpbgpgYGAKCkV4dHJhY3QgdGhlIHBvc3RlcmlvciBkcmF3cwoKYGBge3J9CmRyYXdzIDwtIGFzX2RyYXdzX2RmKGZpdF9iaW4pCmBgYAoKV2UgY2FuIGdldCBzdW1tYXJ5IGluZm9ybWF0aW9uIHVzaW5nIHN1bW1hcmlzZV9kcmF3cygpCgpgYGB7cn0KZHJhd3MgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J2JfSW50ZXJjZXB0JykgfD4KICBzdW1tYXJpc2VfZHJhd3MoKQpgYGAKCldlIGNhbiBjb21wdXRlIHRoZSBwcm9iYWJpbGl0eSBvZiBzdWNjZXNzIGJ5IHVzaW5nIHBsb2dpcyB3aGljaCBpcwplcXVhbCB0byBpbnZlcnNlLWxvZ2l0IGZ1bmN0aW9uCgpgYGB7cn0KZHJhd3MgPC0gZHJhd3MgfD4KICBtdXRhdGVfdmFyaWFibGVzKHRoZXRhPXBsb2dpcyhiX0ludGVyY2VwdCkpCmBgYAoKU3VtbWFyeSBvZiB0aGV0YSBieSB1c2luZyBzdW1tYXJpc2VfZHJhd3MoKQoKYGBge3J9CmRyYXdzIHw+CiAgc3Vic2V0X2RyYXdzKHZhcmlhYmxlPSd0aGV0YScpIHw+CiAgc3VtbWFyaXNlX2RyYXdzKCkKYGBgCgpIaXN0b2dyYW0gb2YgdGhldGEKCmBgYHtyfQptY21jX2hpc3QoZHJhd3MsIHBhcnM9J3RoZXRhJykgKwogIHhsYWIoJ3RoZXRhJykgKwogIHhsaW0oYygwLDEpKQpgYGAKCiMgQ29tcGFyaXNvbiBvZiB0d28gZ3JvdXBzIHdpdGggQmlub21pYWwgCgpBbiBleHBlcmltZW50IHdhcyBwZXJmb3JtZWQgdG8gZXN0aW1hdGUgdGhlIGVmZmVjdCBvZiBiZXRhLWJsb2NrZXJzCm9uIG1vcnRhbGl0eSBvZiBjYXJkaWFjIHBhdGllbnRzLiBBIGdyb3VwIG9mIHBhdGllbnRzIHdlcmUgcmFuZG9tbHkKYXNzaWduZWQgdG8gdHJlYXRtZW50IGFuZCBjb250cm9sIGdyb3VwczoKCi0gb3V0IG9mIDY3NCBwYXRpZW50cyByZWNlaXZpbmcgdGhlIGNvbnRyb2wsIDM5IGRpZWQKLSBvdXQgb2YgNjgwIHJlY2VpdmluZyB0aGUgdHJlYXRtZW50LCAyMiBkaWVkCgpEYXRhLCB3aGVyZSBgZ3JwMmAgaXMgYW4gaW5kaWNhdG9yIHZhcmlhYmxlIGRlZmluZWQgYXMgYSBmYWN0b3IKdHlwZSwgd2hpY2ggaXMgdXNlZnVsIGZvciBjYXRlZ29yaWNhbCB2YXJpYWJsZXMuCgpgYGB7cn0KZGF0YV9iaW4yIDwtIGRhdGEuZnJhbWUoTiA9IGMoNjc0LCA2ODApLAogICAgICAgICAgICAgICAgICAgICAgICB5ID0gYygzOSwyMiksCiAgICAgICAgICAgICAgICAgICAgICAgIGdycDIgPSBmYWN0b3IoYygnY29udHJvbCcsJ3RyZWF0bWVudCcpKSkKYGBgCgpUbyBhbmFseXNlIHdoZXRoZXIgdGhlIHRyZWF0bWVudCBpcyB1c2VmdWwsIHdlIGNhbiB1c2UgQmlub21pYWwKbW9kZWwgZm9yIGJvdGggZ3JvdXBzIGFuZCBjb21wdXRlIG9kZHMtcmF0aW8uIFRvIHJlY3JlYXRlIHRoZSBtb2RlbAphcyB0d28gaW5kZXBlbmRlbnQgKHNlcGFyYXRlKSBiaW5vbWlhbCBtb2RlbHMsIHdlIHVzZSBmb3JtdWxhIGB5IHwKdHJpYWxzKE4pIH4gMCArIGdycDJgLCB3aGljaCBjb3JyZXNwb25kcyB0byBhIG1vZGVsCiRcbWF0aHJte2xvZ2l0fShcdGhldGEpID0gXGFscGhhIFx0aW1lcyAwICsKXGJldGFfXG1hdGhybXtjb250cm9sfVx0aW1lcyB4X1xtYXRocm17Y29udHJvbH0gKwpcYmV0YV9cbWF0aHJte3RyZWF0bWVudH1cdGltZXMgeF9cbWF0aHJte3RyZWF0bWVudH0gPQpcYmV0YV9cbWF0aHJte2NvbnRyb2x9XHRpbWVzIHhfXG1hdGhybXtjb250cm9sfSArClxiZXRhX1xtYXRocm17dHJlYXRtZW50fVx0aW1lcyB4X1xtYXRocm17dHJlYXRtZW50fSQsIHdoZXJlCiR4X1xtYXRocm17Y29udHJvbH0kIGlzIGEgdmVjdG9yIHdpdGggMSBmb3IgY29udHJvbCBhbmQgMCBmb3IKdHJlYXRtZW50LCBhbmQgJHhfXG1hdGhybXt0cmVhdGVtbnR9JCBpcyBhIHZlY3RvciB3aXRoIDEgZm9yCnRyZWF0ZW1udCBhbmQgMCBmb3IgY29udHJvbC4gQXMgb25seSBvZiB0aGUgdmVjdG9ycyBoYXZlIDEsIHRoaXMKY29ycmVzcG9uZHMgdG8gc2VwYXJhdGUgbW9kZWxzCiRcbWF0aHJte2xvZ2l0fShcdGhldGFfXG1hdGhybXtjb250cm9sfSkgPSBcYmV0YV9cbWF0aHJte2NvbnRyb2x9JAphbmQgJFxtYXRocm17bG9naXR9KFx0aGV0YV9cbWF0aHJte3RyZWF0bWVudH0pID0KXGJldGFfXG1hdGhybXt0cmVhdG1lbnR9JC4gIFdlIGNhbiBwcm92aWRlIHRoZSBzYW1lIHByaW9yIGZvciBhbGwKJFxiZXRhJCdzIGJ5IHNldHRpbmcgdGhlIHByaW9yIHdpdGggYGNsYXNzPSdiJ2AuIFdpdGggcHJpb3IKYHN0dWRlbnRfdCg3LCAwLDEuNSlgLCBib3RoICRcYmV0YSQncyBhcmUgc2hydW5rIHRvd2FyZHMgMCwgYnV0CmluZGVwZW5kZW50bHkuCgpgYGB7cn0KZml0X2JpbjIgPC0gYnJtKHkgfCB0cmlhbHMoTikgfiAwICsgZ3JwMiwgZmFtaWx5ID0gYmlub21pYWwoKSwgZGF0YSA9IGRhdGFfYmluMiwKICAgICAgICAgICAgICAgIHByaW9yID0gcHJpb3Ioc3R1ZGVudF90KDcsIDAsMS41KSwgY2xhc3M9J2InKSwKICAgICAgICAgICAgICAgIHNlZWQgPSBTRUVELCByZWZyZXNoID0gMCkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuIGJybXMgaXMgdXNpbmcKdGhlIGZpcnN0IGZhY3RvciBsZXZlbCBgY29udHJvbGAgYXMgdGhlIGJhc2VsaW5lIGFuZCB0aHVzIHJlcG9ydHMKdGhlIGNvZWZmaWNpZW50IChwb3B1bGF0aW9uLWxldmVsIGVmZmVjdCkgZm9yIGB0cmVhdG1lbnRgIChzaG93biBzCmBncnAydHJlYXRtZW50YCkKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLiBXaXRoIGB+IDAgKwpncnAyYCB0aGVyZSBpcyBubyBgSW50ZXJjZXB0YCBhbmQgXGJldGFfXG1hdGhybXtjb250cm9sfSBhbmQKXGJldGFfXG1hdGhybXt0cmVhdG1lbnR9IGFyZSBwcmVzZW50ZWQgYXMgYGdycDJjb250cm9sYCBhbmQKYGdycDJ0cmVhdG1lbnRgLgoKYGBge3J9CmZpdF9iaW4yCmBgYAoKQ29tcHV0ZSB0aGV0YSBmb3IgZWFjaCBncm91cCBhbmQgdGhlIG9kZHMtcmF0aW8uIGBicm1zYCB1c2VzCmJhcmlhYmxlIG5hbWVzIGBiX2dycDJjb250cm9sYCBhbmQgYGJfZ3JwMnRyZWF0bWVudGAgZm9yCiRcYmV0YV9cbWF0aHJte2NvbnRyb2x9JCBhbmQgJFxiZXRhX1xtYXRocm17dHJlYXRtZW50fSQKcmVzcGVjdGl2ZWx5LgoKYGBge3J9CmRyYXdzX2JpbjIgPC0gYXNfZHJhd3NfZGYoZml0X2JpbjIpIHw+CiAgbXV0YXRlKHRoZXRhX2NvbnRyb2wgPSBwbG9naXMoYl9ncnAyY29udHJvbCksCiAgICAgICAgIHRoZXRhX3RyZWF0bWVudCA9IHBsb2dpcyhiX2dycDJ0cmVhdG1lbnQpLAogICAgICAgICBvZGRzcmF0aW8gPSAodGhldGFfdHJlYXRtZW50LygxLXRoZXRhX3RyZWF0bWVudCkpLyh0aGV0YV9jb250cm9sLygxLXRoZXRhX2NvbnRyb2wpKSkKYGBgCgpQbG90IG9kZHNyYXRpbwoKYGBge3J9Cm1jbWNfaGlzdChkcmF3c19iaW4yLCBwYXJzPSdvZGRzcmF0aW8nKSArCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcz1zZXEoMC4yLDEuNixieT0wLjIpKSsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQ9MSwgbGluZXR5cGU9J2Rhc2hlZCcpCmBgYAoKUHJvYmFiaWxpdHkgdGhhdCB0aGUgb2Rkc3JhdGlvPDEKCmBgYHtyfQpkcmF3c19iaW4yIHw+CiAgbXV0YXRlKHBvZGRzcmF0aW8gPSBvZGRzcmF0aW88MSkgfD4KICBzdWJzZXQodmFyaWFibGU9J3BvZGRzcmF0aW8nKSB8PgogIHN1bW1hcmlzZV9kcmF3cyhtZWFuLCBtY3NlX21lYW4pCmBgYAoKb2Rkc3JhdGlvIDk1JSBwb3N0ZXJpb3IgaW50ZXJ2YWwKCmBgYHtyfQpkcmF3c19iaW4yIHw+CiAgc3Vic2V0KHZhcmlhYmxlPSdvZGRzcmF0aW8nKSB8PgogIHN1bW1hcmlzZV9kcmF3cyh+cXVhbnRpbGUoLngsIHByb2JzID0gYygwLjAyNSwgMC45NzUpKSwgfm1jc2VfcXVhbnRpbGUoLngsIHByb2JzID0gYygwLjAyNSwgMC45NzUpKSkKYGBgCgpNYWtlIHByaW9yIHNlbnNpdGl2aXR5IGFuYWx5c2lzIGJ5IHBvd2VyLXNjYWxpbmcgYm90aCBwcmlvciBhbmQKbGlrZWxpaG9vZC4gIEZvY3VzIG9uIG9kZHNyYXRpbyB3aGljaCBpcyB0aGUgcXVhbnRpdHkgb2YKaW50ZXJlc3QuIFdlIHNlZSB0aGF0IHRoZSBsaWtlbGlob29kIGlzIG11Y2ggbW9yZSBpbmZvcm1hdGl2ZSB0aGFuCnRoZSBwcmlvciwgYW5kIHdlIHdvdWxkIGV4cGVjdCB0byBzZWUgYSBkaWZmZXJlbnQgcG9zdGVyaW9yIG9ubHkKd2l0aCBhIGhpZ2hseSBpbmZvcm1hdGl2ZSBwcmlvciAocG9zc2libHkgYmFzZWQgb24gcHJldmlvdXMgc2ltaWxhcgpleHBlcmltZW50cykuCgpgYGB7cn0Kb2Rkc3JhdGlvIDwtIGRyYXdzX2JpbjIgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J29kZHNyYXRpbycpCmBgYAoKUHJpb3IgYW5kIGxpa2VsaWhvb2Qgc2Vuc2l0aXZpdHkgcGxvdCBzaG93cyBwb3N0ZXJpb3IgZGVuc2l0eSBlc3RpbWF0ZQpkZXBlbmRpbmcgb24gYW1vdW50IG9mIHBvd2VyLXNjYWxpbmcuIE92ZXJsYXBwaW5nIGxpbmUgaW5kaWNhdGUgbG93CnNlbnNpdGl2aXR5IGFuZCB3aWRlciBnYXBzIGJldHdlZW4gbGluZSBpbmRpY2F0ZSBncmVhdGVyIHNlbnNpdGl2aXR5LgoKYGBge3J9CnBvd2Vyc2NhbGVfc2VxdWVuY2UoZml0X2JpbjIsIHByZWRpY3Rpb24gPSBcKHgsIC4uLikgb2Rkc3JhdGlvKSB8PgogIHBvd2Vyc2NhbGVfcGxvdF9kZW5zKHZhcmlhYmxlcz0nb2Rkc3JhdGlvJykgKwogICMgc3dpdGNoIHJvd3MgYW5kIGNvbHMKICBmYWNldF9ncmlkKHJvd3M9dmFycyguZGF0YSR2YXJpYWJsZSksCiAgICAgICAgICAgICBjb2xzPXZhcnMoLmRhdGEkY29tcG9uZW50KSkgKwogICMgY2xlYW5pbmcKICBnZ3RpdGxlKE5VTEwsTlVMTCkgKwogIGxhYnMoeD0nT2Rkcy1yYXRpbycsIHk9TlVMTCkgKwogIHNjYWxlX3lfY29udGludW91cyhicmVha3M9TlVMTCkgKwogIHRoZW1lKGF4aXMubGluZS55PWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBzdHJpcC50ZXh0Lnk9ZWxlbWVudF9ibGFuaygpKSArCiAgIyByZWZlcmVuY2UgbGluZQogIGdlb21fdmxpbmUoeGludGVyY2VwdD0xLCBsaW5ldHlwZT0nZGFzaGVkJykKYGBgCgpXZSBjYW4gc3VtbWFyaXNlIHRoZSBwcmlvciBhbmQgbGlrZWxpaG9vZCBzZW5zaXRpdml0eSB1c2luZwpjdW11bGF0aXZlIEplbnNlbi1TaGFubm9uIGRpc3RhbmNlLgoKYGBge3J9CnBvd2Vyc2NhbGVfc2Vuc2l0aXZpdHkoZml0X2JpbjIsIHByZWRpY3Rpb24gPSBcKHgsIC4uLikgb2Rkc3JhdGlvLCBudW1fYXJncz1saXN0KGRpZ2l0cz0yKQogICAgICAgICAgICAgICAgICAgICAgICkkc2Vuc2l0aXZpdHkgfD4KICAgICAgICAgICAgICAgICAgICAgICAgIGZpbHRlcih2YXJpYWJsZT09J29kZHNyYXRpbycpIHw+CiAgICAgICAgICAgICAgICAgICAgICAgICBtdXRhdGUoYWNyb3NzKHdoZXJlKGlzLmRvdWJsZSksICB+bnVtKC54LCBkaWdpdHM9MikpKQpgYGAKCkFib3ZlIHdlIHVzZWQgZm9ybXVsYSBgeSB8IHRyaWFscyhOKSB+IDAgKyBncnAyYCB0byBoYXZlIHNlcGFyYXRlCm1vZGVsIGZvciBjb250cm9sIGFuZCB0cmVhdG1lbnQgZ3JvdXAuIEFuIGFsdGVybmF0aXZlIG1vZGVsIGB5IHwKdHJpYWxzKE4pIH4gZ3JwMmAgd2hpY2ggaXMgZXF1YWwgdG8gYHkgfCB0cmlhbHMoTikgfiAxICsgZ3JwMmAsCndvdWxkIGNvcnJlc3BvbmQgdG8gYSBtb2RlbCAkXG1hdGhybXtsb2dpdH0oXHRoZXRhKSA9IFxhbHBoYSBcdGltZXMKMSArIFxiZXRhX1xtYXRocm17dHJlYXRtZW50fVx0aW1lcyB4X1xtYXRocm17dHJlYXRtZW50fSA9IFxhbHBoYSArClxiZXRhX1xtYXRocm17dHJlYXRtZW50fVx0aW1lcyB4X1xtYXRocm17dHJlYXRtZW50fS4gTm93ICRcYWxwaGEkCm1vZGVscyB0aGUgcHJvYmFiaWxpdHkgb2YgZGVhdGggKHZpYSBsb2dpc3RpYyBsaW5rKSBpbiB0aGUgY29udHJvbApncm91cCBhbmQgJFxhbHBoYSArIFxiZXRhX1xtYXRocm17dHJlYXRtZW50fSQgbW9kZWxzIHRoZQpwcm9iYWJpbGl0eSBvZiBkZWF0aCAodmlhIGxvZ2lzdGljIGxpbmspIGluIHRoZSB0cmVhdG1lbnQKZ3JvdXAuIE5vdyB0aGUgbW9kZWxzIGZvciB0aGUgZ3JvdXBzIGFyZSBjb25uZWN0ZWQuIEZ1cnRoZXJtb3JlLCBpZgp3ZSBzZXQgaW5kZXBlbmRlbnQgYHN0dWRlbnRfdCg3LCAwLCAxLjUpYCBwcmlvcnMgb24gJFxhbHBoYSQgYW5kCiRcYmV0YV9cbWF0aHJte3RyZWF0bWVudH0kLCB0aGUgaW1wbGllZCBwcmlvcnMgb24KJFx0aGV0YV9cbWF0aHJte2NvbnRyb2x9JCBhbmQgJFx0aGV0YV9cbWF0aHJte3RyZWF0bWVudH0kIGFyZQpkaWZmZXJlbnQuIFdlIGNhbiB2ZXJpZnkgdGhpcyB3aXRoIGEgcHJpb3Igc2ltdWxhdGlvbi4KCgpgYGB7cn0KZGF0YS5mcmFtZSh0aGV0YV9jb250cm9sID0gcGxvZ2lzKGdnZGlzdDo6cnN0dWRlbnRfdChuPTIwMDAwLCBkZj03LCBtdT0wLCBzaWdtYT0xLjUpKSkgfD4KICBtY21jX2hpc3QoKSArCiAgeGxpbShjKDAsMSkpICsKICBsYWJzKHRpdGxlPSdzdHVkZW50X3QoNywgMCwgMS41KSBwcmlvciBvbiBJbnRlcmNlcHQnKSArCmRhdGEuZnJhbWUodGhldGFfdHJlYXRtZW50ID0gcGxvZ2lzKGdnZGlzdDo6cnN0dWRlbnRfdChuPTIwMDAwLCBkZj03LCBtdT0wLCBzaWdtYT0xLjUpKSsKICAgICAgICAgICAgIHBsb2dpcyhnZ2Rpc3Q6OnJzdHVkZW50X3Qobj0yMDAwMCwgZGY9NywgbXU9MCwgc2lnbWE9MS41KSkpIHw+CiAgbWNtY19oaXN0KCkgKwogIHhsaW0oYygwLDEpKSArCiAgbGFicyh0aXRsZT0nc3R1ZGVudF90KDcsIDAsIDEuNSkgcHJpb3Igb24gSW50ZXJjZXB0IGFuZCBiX2dycDJ0cmVhdG1lbnQnKQpgYGAKCkluIHRoaXMgY2FzZSwgd2l0aCByZWxhdGl2ZWx5IGJpZyB0cmVhdG1lbnQgYW5kIGNvbnRyb2wgZ3JvdXAsIHRoZQpsaWtlbGlob29kIGlzIGluZm9ybWF0aXZlLCBhbmQgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiB1c2luZyBgeSB8CnRyaWFscyhOKSB+IDAgKyBncnAyYCBvciBgeSB8IHRyaWFscyhOKSB+IGdycDJgIGlzIG5lZ2xpZ2libGUuCgpUaGlyZCBvcHRpb24gd291bGQgYmUgYSBoaWVyYXJjaGljYWwgbW9kZWwgd2l0aCBmb3JtdWxhIGB5IHwKdHJpYWxzKE4pIH4gMSArICgxIHwgZ3JwMilgLCB3aGljaCBpcyBlcXVpdmFsZW50IHRvIGB5IHwgdHJpYWxzKE4pCn4gMSArICgxIHwgZ3JwMilgLCBhbmQgY29ycmVzcG9uZHMgdG8gYSBtb2RlbAokXG1hdGhybXtsb2dpdH0oXHRoZXRhKSA9IFxhbHBoYSBcdGltZXMgMSArClxiZXRhX1xtYXRocm17Y29udHJvbH1cdGltZXMgeF9cbWF0aHJte2NvbnRyb2x9ICsKXGJldGFfXG1hdGhybXt0cmVhdG1lbnR9XHRpbWVzIHhfXG1hdGhybXt0cmVhdG1lbnR9JCwgYnV0IG5vdyB0aGUKcHJpb3Igb24gJFxiZXRhX1xtYXRocm17Y29udHJvbH0kIGFuZCAkXGJldGFfXG1hdGhybXt0cmVhdG1lbnR9JCBpcwokXG1hdGhybXtub3JtYWx9KDAsIFxzaWdtYV9cbWF0aHJte2dycH0pJC4gVGhlIGRlZmF1bHQgYGJybXNgIHByaW9yCmZvciAkXHNpZ21hX1xtYXRocm17Z3JwfSQgaXMgYHN0dWRlbnRfdCgzLCAwLCAyLjUpYC4gTm93ICRcYWxwaGEkCm1vZGVscyB0aGUgb3ZlcmFsbCBwcm9iYWJsaXR5IG9mIGRlYXRoICh2aWEgbG9naXN0aWMgbGluayksIGFuZAokXGJldGFfXG1hdGhybXtjb250cm9sfSQgYW5kICRcYmV0YV9cbWF0aHJte3RyZWF0bWVudH0kIG1vZGVsIHRoZQpkaWZmZXJlbmNlIGZyb20gdGhhdCBoYXZpbmcgdGhlIHNhbWUgcHJpb3IuIFByaW9yIGZvcgokXGJldGFfXG1hdGhybXtjb250cm9sfSQgYW5kICRcYmV0YV9cbWF0aHJte3RyZWF0bWVudH0kIGluY2x1ZGVzCnVua25vd24gc2NhbGUgJFxzaWdtYV9cbWF0aHJte2dycH0kLiBJZiB0aGUgdGhlcmUgaXMgbm90IGRpZmZlcmVuY2UKYmV0d2VlbiBjb250cm9sIGFuZCB0cmVhdG1lbnQgZ3JvdXBzLCB0aGUgcG9zdGVyaW9yIG9mCiRcc2lnbWFfXG1hdGhybXtncnB9JCBoYXMgbW9yZSBtYXNzIG5lYXIgMCwgYW5kIGJpZ2dlciB0aGUKZGlmZmVyZW5jZSBiZXR3ZWVuIGNvbnRyb2wgYW5kIHRyZWF0bWVudCBncm91cHMgYXJlLCBtb3JlIG1hc3MKdGhlcmUgaXMgYXdheSBmcm9tIDAuIFdpdGgganVzdCB0d28gZ3JvdXBzLCB0aGVyZSBpcyBub3QgbXVjaAppbmZvcm1hdGlvbiBhYm91dCAkXHNpZ21hX1xtYXRocm17Z3JwfSQsIGFuZCB1bmxlc3MgdGhlcmUgaXMgYQppbmZvcm1hdGl2ZSBwcmlvciBvbiAkXHNpZ21hX1xtYXRocm17Z3JwfSQsIHR3byBncm91cCBoaWVyYXJjaGljYWwKbW9kZWwgaXMgbm90IHRoYXQgdXNlZnVsLiBIaWVyYXJjaGljYWwgbW9kZWxzIGFyZSBtb3JlIHVzZWZ1bCB3aXRoCm1vcmUgdGhhbiB0d28gZ3JvdXBzLiBJbiB0aGUgZm9sbG93aW5nLCB3ZSB1c2UgdGhlIHByZXZpb3VzbHkgdXNlZApgc3R1ZGVudF90KDcsIDAsMS41KWAgcHJpb3Igb24gaW50ZXJjZXB0IGFuZCB0aGUgZGVmYXVsdCBgYnJtc2AKcHJpb3IgYHN0dWRlbnRfdCgzLCAwLCAyLjUpYCBvbiAkXHNpZ21hX1xtYXRocm17Z3JwfSQuCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X2JpbjIgPC0gYnJtKHkgfCB0cmlhbHMoTikgfiAxICsgKDEgfCBncnAyKSwgZmFtaWx5ID0gYmlub21pYWwoKSwgZGF0YSA9IGRhdGFfYmluMiwKICAgICAgICAgICAgICAgIHByaW9yID0gcHJpb3Ioc3R1ZGVudF90KDcsIDAsMS41KSwgY2xhc3M9J0ludGVyY2VwdCcpLAogICAgICAgICAgICAgICAgc2VlZCA9IFNFRUQsIHJlZnJlc2ggPSAwLCBjb250cm9sPWxpc3QoYWRhcHRfZGVsdGE9MC45OSkpCmBgYAoKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLiBUaGUgc3VtbWFyeQpyZXBvcnRzIHRoYXQgdGhlcmUgYXJlIEdyb3VwLUxldmVsIEVmZmVjdHM6IGB+Z3JwMmAgd2l0aCAyIGxldmVscwooY29udHJvbCBhbmQgdHJlYXRtZW50KSwgd2l0aCBgc2QoSW50ZXJjZXB0KWAgZGVub3RpbmcKJFxzaWdtYV9cbWF0aHJte2dycH0kLiBJbiBhZGRpdGlvbiwgdGhlIHN1bW1hcnkgbGlzdHMKUG9wdWxhdGlvbi1MZXZlbCBFZmZlY3RzOiBgSW50ZXJjZXB0YCAoJFxhbHBoYSQpIGFzIGluIHRoZSBwcmV2b3VzCm5vbi1oaWVyYXJjaGljYWwgbW9kZWxzLgoKYGBge3J9CmZpdF9iaW4yCmBgYAoKV2UgY2FuIGFsc28gbG9vayBhdCB0aGUgdmFyaWFibGUgbmFtZXMgYGJybXNgIHVzZXMgaW50ZXJuYWxseQoKYGBge3J9CmFzX2RyYXdzX3J2YXJzKGZpdF9iaW4yKQpgYGAKCkFsdGhvdWdoIHRoZXJlIGlzIG5vIGRpZmZlcmVuY2UsIGlsbHVzdHJhdGUgaG93IHRvIGNvbXB1dGUgdGhlCm9kZHNyYXRpbyBmcm9tIGhpZXJhcmNoaWNhbCBtb2RlbAoKYGBge3J9CmRyYXdzX2JpbjIgPC0gYXNfZHJhd3NfZGYoZml0X2JpbjIpCm9kZHNyYXRpbyA8LSBkcmF3c19iaW4yIHw+CiAgbXV0YXRlX3ZhcmlhYmxlcyh0aGV0YV9jb250cm9sID0gcGxvZ2lzKGJfSW50ZXJjZXB0ICsgYHJfZ3JwMltjb250cm9sLEludGVyY2VwdF1gKSwKICAgICAgICAgICAgICAgICAgIHRoZXRhX3RyZWF0bWVudCA9IHBsb2dpcyhiX0ludGVyY2VwdCArIGByX2dycDJbdHJlYXRtZW50LEludGVyY2VwdF1gKSwKICAgICAgICAgICAgICAgICAgIG9kZHNyYXRpbyA9ICh0aGV0YV90cmVhdG1lbnQvKDEtdGhldGFfdHJlYXRtZW50KSkvKHRoZXRhX2NvbnRyb2wvKDEtdGhldGFfY29udHJvbCkpKSB8PgogIHN1YnNldF9kcmF3cyh2YXJpYWJsZT0nb2Rkc3JhdGlvJykKb2Rkc3JhdGlvIHw+IG1jbWNfaGlzdCgpICsKICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzPXNlcSgwLjIsMS42LGJ5PTAuMikpKwogIGdlb21fdmxpbmUoeGludGVyY2VwdD0xLCBsaW5ldHlwZT0nZGFzaGVkJykKYGBgCgpNYWtlIGFsc28gcHJpb3Igc2Vuc2l0aXZpdHkgYW5hbHlzaXMgd2l0aCBmb2N1cyBvbiBvZGRzcmF0aW8uCgpgYGB7cn0KcG93ZXJzY2FsZV9zZW5zaXRpdml0eShmaXRfYmluMiwgcHJlZGljdGlvbiA9IFwoeCwgLi4uKSBvZGRzcmF0aW8sIG51bV9hcmdzPWxpc3QoZGlnaXRzPTIpCiAgICAgICAgICAgICAgICAgICAgICAgKSRzZW5zaXRpdml0eSB8PgogICAgICAgICAgICAgICAgICAgICAgICAgZmlsdGVyKHZhcmlhYmxlPT0nb2Rkc3JhdGlvJykgfD4KICAgICAgICAgICAgICAgICAgICAgICAgIG11dGF0ZShhY3Jvc3Mod2hlcmUoaXMuZG91YmxlKSwgIH5udW0oLngsIGRpZ2l0cz0yKSkpCmBgYAoKIyBMaW5lYXIgR2F1c3NpYW4gbW9kZWwKClVzZSB0aGUgS2lscGlzasOkcnZpIHN1bW1lciBtb250aCB0ZW1wZXJhdHVyZXMgMTk1Mi0tMjAyMiBkYXRhIGZyb20gYGFhbHRvYmRhYCBwYWNrYWdlCgpgYGB7cn0KbG9hZCh1cmwoJ2h0dHBzOi8vZ2l0aHViLmNvbS9hdmVodGFyaS9CREFfY291cnNlX0FhbHRvL3Jhdy9tYXN0ZXIvcnBhY2thZ2UvZGF0YS9raWxwaXNqYXJ2aTIwMjIucmRhJykpCmRhdGFfbGluIDwtIGRhdGEuZnJhbWUoeWVhciA9IGtpbHBpc2phcnZpMjAyMiR5ZWFyLAogICAgICAgICAgICAgICAgICAgICAgIHRlbXAgPSBraWxwaXNqYXJ2aTIwMjIkdGVtcC5zdW1tZXIpCmBgYAoKUGxvdCB0aGUgZGF0YQoKYGBge3J9CmRhdGFfbGluIHw+CiAgZ2dwbG90KGFlcyh5ZWFyLCB0ZW1wKSkgKwogIGdlb21fcG9pbnQoY29sb3I9MikgKwogIGxhYnMoeD0gIlllYXIiLCB5ID0gJ1N1bW1lciB0ZW1wLiBAS2lscGlzasOkcnZpJykgKwogIGd1aWRlcyhsaW5ldHlwZSA9ICJub25lIikKYGBgCgpUbyBhbmFseXNlIGhhcyB0aGVyZSBiZWVuIGNoYW5nZSBpbiB0aGUgYXZlcmFnZSBzdW1tZXIgbW9udGgKdGVtcGVyYXR1cmUgd2UgdXNlIGEgbGluZWFyIG1vZGVsIHdpdGggR2F1c3NpYW4gbW9kZWwgZm9yIHRoZQp1bmV4cGxhaW5lZCB2YXJpYXRpb24uIEJ5IGRlZmF1bHQgYnJtcyB1c2VzIHVuaWZvcm0gcHJpb3IgZm9yIHRoZQpjb2VmZmljaWVudHMuCgpGb3JtdWxhIGB0ZW1wIH4geWVhcmAgY29ycmVzcG9uZHMgdG8gbW9kZWwgJFxtYXRocm17dGVtcH0gfgpcbWF0aHJte25vcm1hbH0oXGFscGhhICsgXGJldGEgXHRpbWVzIFxtYXRocm17dGVtcH0sIFxzaWdtYSkuICBUaGUKbW9kZWwgY291bGQgYWxzbyBiZSBkZWZpbmVkIGFzIGB0ZW1wIH4gMSArIHllYXJgIHdoaWNoIGV4cGxpY2l0bHkKc2hvd3MgdGhlIGludGVyY2VwdCAoJFxhbHBoYSQpIHBhcnQuIFVzaW5nIHRoZSB2YXJpYWJsZSBuYW1lcwpgYnJtc2AgdXNlcyB0aGUgbW9kZWwgY2FuIGJlIHdyaXR0ZW4gYWxzbyBhcyBgdGVtcCB+Cm5vcm1hbChiX0ludGVyY2VwdCoxICsgYl95ZWFyKnllYXIsIHNpZ21hKWAuIFdlIHN0YXJ0IHdpdGggdGhlCmRlZmF1bHQgcHJpb3JzIHRvIHNlZSBzb21lIHRyaWNrcyB0aGF0IGBicm1zYCBkb2VzIGJlaGluZCB0aGUKY3VydGFpbi4KCmBgYHtyfQpmaXRfbGluIDwtIGJybSh0ZW1wIH4geWVhciwgZGF0YSA9IGRhdGFfbGluLCBmYW1pbHkgPSBnYXVzc2lhbigpLAogICAgICAgICAgICAgICBzZWVkID0gU0VFRCwgcmVmcmVzaCA9IDApCmBgYAoKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLgoKYGBge3J9CmZpdF9saW4KYGBgCgpDb252ZXJnZW5jZSBkaWFnbm9zdGljcyBsb29rIGdvb2QuIFdlIHNlZSB0aGF0IHBvc3RlcmlvciBtZWFuIG9mCmBJbnRlcmNlcHRgIGlzIC0zNC43LCB3aGljaCBtYXkgc291bmQgc3RyYW5nZSwgYnV0IHRoYXQgaXMgdGhlCmludGVyY2VwdCBhdCB5ZWFyIDAsIHRoYXQgaXMsIHZlcnkgZmFyIGZyb20gdGhlIGRhdGEgcmFuZ2UsIGFuZAp0aHVzIGRvZXNuJ3QgaGF2ZSBtZWFuaW5nZnVsIGludGVycHJldGF0aW9uIGRpcmVjdGx5LiBUaGUgcG9zdGVyaW9yCm1lYW4gb2YgYHllYXJgIGNvZWZmaWNpZW50IGlzIDAuMDIsIHRoYXQgaXMsIHdlIGVzdGltYXRlIHRoYXQgdGhlCnN1bW1lciB0ZW1wZXJhdHVyZSBpcyBpbmNyZWFzaW5nIDAuMDLCsEMgcGVyIHllYXIgKHdoaWNoIHdvdWxkIG1ha2UKMcKwQyBpbiA1MCB5ZWFycykuCgpXZSBjYW4gY2hlY2sgJFJeMiQgd2hpY2ggY29ycmVzcG9uZHMgdG8gdGhlIHByb3BvcmlvbiBvZiB2YXJpYW5jZQpleHBsYWluZWQgYnkgdGhlIG1vZGVsLiBUaGUgbGluZWFyIG1vZGVsIGV4cGxhaW5zIDAuMTY9MTYlIG9mIHRoZQp0b3RhbCBkYXRhIHZhcmlhbmNlLgoKYGBge3J9CmJheWVzX1IyKGZpdF9saW4pIHw+IHJvdW5kKDIpCmBgYAoKV2UgY2FuIGNoZWNrIHRoZSBhbGwgdGhlIHByaW9ycyB1c2VkLiAKCmBgYHtyfQpwcmlvcl9zdW1tYXJ5KGZpdF9saW4pCmBgYAoKV2Ugc2VlIHRoYXQgYGNsYXNzPWJgIGFuZCBgY29lZj15ZWFyYCBoYXZlIGBmbGF0YCwgdGhhdCBpcywKaW1wcm9wZXIgdW5pZm9ybSBwcmlvciwgYEludGVyY2VwdGAgaGFzIGBzdHVkZW50X3QoMywgOS41LCAyLjUpYCwKYW5kIGBzaWdtYWAgaGFzIGBzdHVkZW50X3QoMywgMCwgMi41KWAgcHJpb3IuICBJbiBnZW5lcmFsIGl0IGlzCmdvb2QgdG8gdXNlIHByb3BlciBwcmlvcnMsIGJ1dCBzb21ldGltZXMgZmxhdCBwcmlvcnMgYXJlIGZpbmUgYW5kCnByb2R1Y2UgcHJvcGVyIHBvc3RlcmlvciAobGlrZSBpbiB0aGlzIGNhc2UpLiBJbXBvcnRhbnQgcGFydCBoZXJlCmlzIHRoYXQgYnkgZGVmYXVsdCwgYGJybXNgIHNldHMgdGhlIHByaW9yIG9uIEludGVyY2VwdCBhZnRlcgpjZW50ZXJpbmcgdGhlIGNvdmFyaWF0ZSB2YWx1ZXMgKGRlc2lnbiBtYXRyaXgpLiBJbiB0aGlzIGNhc2UsCmBicm1zYCB1c2VzIGB0ZW1wIC0gbWVhbih0ZW1wKSA9IHRlbXAgLSAxOTg3YCBpbnN0ZWFkIG9mIG9yaWdpbmFsCnllYXJzLiBUaGlzIGluIGdlbmVyYWwgaW1wcm92ZXMgdGhlIHNhbXBsaW5nIGVmZmljaWVuY3kuIEFzIHRoZQpgSW50ZXJjZXB0YCBpcyBub3cgZGVmaW5lZCBhdCB0aGUgbWlkZGxlIG9mIHRoZSBkYXRhLCB0aGUgZGVmYXVsdApgSW50ZXJjZXB0YCBwcmlvciBpcyBjZW50ZXJlZCBvbiBtZWRpYW4gb2YgdGhlIHRhcmdldCAoaGVyZSB0YXJnZXQKaXMgYHllYXJgKS4gSWYgd2Ugd291bGQgbGlrZSB0byBzZXQgaW5mb3JtYXRpdmUgcHJpb3JzLCB3ZSBuZWVkIHRvCnNldCB0aGUgaW5mb3JtYXRpdmUgcHJpb3Igb24gYEludGVyY2VwdGAgZ2l2ZW4gdGhlIGNlbnRlcmVkCmNvdmFyaWF0ZSB2YWx1ZXMuIFdlIGNhbiB0dXJuIG9mIHRoZSBjZW50ZXJpbmcgYnkgc2V0dGluZyBhcmd1bWVudApgY2VudGVyPUZBTFNFYCwgYW5kIHdlIGNhbiBzZXQgdGhlIHByaW9yIG9uIG9yaWdpbmFsIGludGVyY2VwdCBieQp1c2luZyBhIGZvcm11bGEgYHRlbXAgfiAwICsgSW50ZXJjZXB0ICsgeWVhcmAuIEluIHRoaXMgY2FzZSwgd2UgYXJlCmhhcHB5IHdpdGggdGhlIGRlZmF1bHQgcHJpb3IgZm9yIHRoZSBpbnRlcmNlcHQuIEluIHRoaXMgc3BlY2lmaWMKY2Fzc2UsIHRoZSBmbGF0IHByaW9yIG9uIGNvZWZmaWNpZW50IGlzIGFsc28gZmluZSwgYnV0IHdlIGFkZCBhbgp3ZWFrbHkgaW5mb3JtYXRpdmUgcHJpb3IganVzdCBmb3IgdGhlIGlsbHVzdHJhdGlvbi4gTGV0J3MgYXNzdW1lIHdlCmV4cGVjdCB0aGUgdGVtcGVyYXR1cmUgdG8gY2hhbmdlIGxlc3MgdGhhbiAxwrBDIGluIDEwIHllYXJzLiBXaXRoCmBzdHVkZW50X3QoMywgMCwgMC4wMylgIGFib3V0IDk1JSBwcmlvciBtYXNzIGhhcyBsZXNzIHRoYW4gMC4xwrBDCmNoYW5nZSBpbiB5ZWFyLCBhbmQgd2l0aCBsb3cgZGVncmVlcyBvZiBmcmVlZG9tICgzKSB3ZSBoYXZlIHRoaWNrCnRhaWxzIG1ha2luZyB0aGUgbGlrZWxpaG9vZCBkb21pbmF0ZSBpbiBjYXNlIG9mIHByaW9yLWRhdGEKY29uZmxpY3QuIEluIHJlYWwgbGlmZSwgd2UgZG8gaGF2ZSBtdWNoIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgdGhlCnRlbXBlcmF0dXJlIGNoYW5nZSwgYW5kIG5hdHVyYWxseSBhIGhpZXJhcmNoaWNhbCBzcGF0aW8tdGVtcG9yYWwKbW9kZWwgd2l0aCBhbGwgdGVtcGVyYXR1cmUgbWVhc3VyZW1lbnQgbG9jYXRpb25zIHdvdWxkIGJlIGV2ZW4KYmV0dGVyLgoKYGBge3J9CmZpdF9saW4gPC0gYnJtKHRlbXAgfiB5ZWFyLCBkYXRhID0gZGF0YV9saW4sIGZhbWlseSA9IGdhdXNzaWFuKCksCiAgICAgICAgICAgICAgIHByaW9yID0gcHJpb3Ioc3R1ZGVudF90KDMsIDAsIDAuMDMpLCBjbGFzcz0nYicpLAogICAgICAgICAgICAgICBzZWVkID0gU0VFRCwgcmVmcmVzaCA9IDApCmBgYAoKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLgoKYGBge3J9CmZpdF9saW4KYGBgCgpNYWtlIHByaW9yIHNlbnNpdGl2aXR5IGFuYWx5c2lzIGJ5IHBvd2VyLXNjYWxpbmcgYm90aCBwcmlvciBhbmQgbGlrZWxpaG9vZC4KCmBgYHtyfQpwb3dlcnNjYWxlX3NlbnNpdGl2aXR5KGZpdF9saW4pJHNlbnNpdGl2aXR5IHw+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbXV0YXRlKGFjcm9zcyh3aGVyZShpcy5kb3VibGUpLCAgfm51bSgueCwgZGlnaXRzPTIpKSkKYGBgCgpPdXIgd2Vha2x5IGluZm9ybWF0aXZlIHByb3BlciBwcmlvciBoYXMgbmVnbGlnaWJsZSBzZW5zaXRpdml0eSwgYW5kCnRoZSBsaWtlbGlob29kIGlzIGluZm9ybWF0aXZlLgpFeHRyYWN0IHRoZSBwb3N0ZXJpb3IgZHJhd3MgYW5kIGNoZWNrIHRoZSBzdW1tYXJpZXMKCmBgYHtyfQpkcmF3c19saW4gPC0gYXNfZHJhd3NfZGYoZml0X2xpbikgCmRyYXdzX2xpbiB8PiBzdW1tYXJpc2VfZHJhd3MoKQpgYGAKCklmIG9uZSBvZiB0aGUgY29sdW1ucyBpcyBoaWRkZW4gd2UgY2FuIGZvcmNlIHByaW50aW5nIGFsbCBjb2x1bW5zCgpgYGB7cn0KZHJhd3NfbGluIHw+IHN1bW1hcmlzZV9kcmF3cygpIHw+IHByaW50KHdpZHRoPUluZikKYGBgCgpIaXN0b2dyYW0gb2YgYl95ZWFyCgpgYGB7cn0KZHJhd3NfbGluIHw+CiAgbWNtY19oaXN0KHBhcnM9J2JfeWVhcicpICsKICB4bGFiKCdBdmVyYWdlIHRlbXBlcmF0dXJlIGluY3JlYXNlIHBlciB5ZWFyJykKYGBgCgpQcm9iYWJpbGl0eSB0aGF0IHRoZSBjb2VmZmljaWVudCBiX3llYXIgPiAwIGFuZCB0aGUgY29ycmVzcG9uZGluZyBNQ1NFCgpgYGB7cn0KZHJhd3NfbGluIHw+CiAgbXV0YXRlKElfYl95ZWFyX2d0XzAgPSBiX3llYXI+MCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J0lfYl95ZWFyX2d0XzAnKSB8PgogIHN1bW1hcmlzZV9kcmF3cyhtZWFuLCBtY3NlX21lYW4pCmBgYAoKQWxsIHBvc3RlcmlvciBkcmF3cyBoYXZlIGBiX3llYXI+MGAsIHRoZSBwcm9iYWJpbGl0eSBnZXRzIHJvdW5kZWQKdG8gMSwgYW5kIE1DU0UgaXMgbm90IGF2YWlsYWJsZSBhcyB0aGUgb2JzZXJldmQgcG9zdGVyaW9yIHZhcmlhbmNlCmlzIDAuCgo5NSUgcG9zdGVyaW9yIGludGVydmFsIGZvciB0ZW1wZXJhdHVyZSBpbmNyZWFzZSBwZXIgMTAwIHllYXJzCgpgYGB7cn0KZHJhd3NfbGluIHw+CiAgbXV0YXRlKGJfeWVhcl8xMDAgPSBiX3llYXIqMTAwKSB8PgogIHN1YnNldF9kcmF3cyh2YXJpYWJsZT0nYl95ZWFyXzEwMCcpIHw+CiAgc3VtbWFyaXNlX2RyYXdzKH5xdWFudGlsZSgueCwgcHJvYnMgPSBjKDAuMDI1LCAwLjk3NSkpLAogICAgICAgICAgICAgICAgICB+bWNzZV9xdWFudGlsZSgueCwgcHJvYnMgPSBjKDAuMDI1LCAwLjk3NSkpLAogICAgICAgICAgICAgICAgICAubnVtX2FyZ3MgPSBsaXN0KGRpZ2l0cyA9IDIsIG5vdGF0aW9uID0gImRlYyIpKQpgYGAKClBsb3QgcG9zdGVyaW9yIGRyYXdzIG9mIHRoZSBsaW5lYXIgZnVuY3Rpb24gdmFsdWVzIGF0IGVhY2ggeWVhci4KYGFkZF9saW5wcmVkX2RyYXdzKClgIHRha2VzIHRoZSB5ZWFycyBmcm9tIHRoZSBkYXRhIGFuZCB1c2VzIGBmaXRfbGluYCB0byBtYWtlCnRoZSBwcmVkaWN0aW9ucy4KCmBgYHtyfQpkYXRhX2xpbiB8PgogIGFkZF9saW5wcmVkX2RyYXdzKGZpdF9saW4pIHw+CiAgIyBwbG90IGRhdGEKICBnZ3Bsb3QoYWVzKHg9eWVhciwgeT10ZW1wKSkgKwogIGdlb21fcG9pbnQoY29sb3I9MikgKwogICMgcGxvdCBsaW5lcmliYm9uIGZvciB0aGUgbGluZWFyIG1vZGVsCiAgc3RhdF9saW5lcmliYm9uKGFlcyh5ID0gLmxpbnByZWQpLCAud2lkdGggPSBjKC45NSksIGFscGhhID0gMS8yLCBjb2xvcj1icmV3ZXIucGFsKDUsICJCbHVlcyIpW1s1XV0pICsKICAjIGRlY29yYXRpb24KICBzY2FsZV9maWxsX2JyZXdlcigpKwogIGxhYnMoeD0gIlllYXIiLCB5ID0gJ1N1bW1lciB0ZW1wLiBAS2lscGlzasOkcnZpJykgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpKwogIHNjYWxlX3hfY29udGludW91cyhicmVha3M9c2VxKDE5NTAsMjAyMCxieT0xMCkpCmBgYAoKQWx0ZXJuYXRpdmVsbHkgcGxvdCBhIHNwYWdoZXR0aSBwbG90IGZvciAxMDAgZHJhd3MKCmBgYHtyfQpkYXRhX2xpbiB8PgogIGFkZF9saW5wcmVkX2RyYXdzKGZpdF9saW4sIG5kcmF3cz0xMDApIHw+CiAgIyBwbG90IGRhdGEKICBnZ3Bsb3QoYWVzKHg9eWVhciwgeT10ZW1wKSkgKwogIGdlb21fcG9pbnQoY29sb3I9MikgKwogICMgcGxvdCBhIGxpbmUgZm9yIGVhY2ggcG9zdGVyaW9yIGRyYXcKICBnZW9tX2xpbmUoYWVzKHk9LmxpbnByZWQsIGdyb3VwPS5kcmF3KSwgYWxwaGEgPSAxLzIsIGNvbG9yID0gYnJld2VyLnBhbCg1LCAiQmx1ZXMiKVtbM11dKSsKICAjIGRlY29yYXRpb24KICBzY2FsZV9maWxsX2JyZXdlcigpKwogIGxhYnMoeD0gIlllYXIiLCB5ID0gJ1N1bW1lciB0ZW1wLiBAS2lscGlzasOkcnZpJykgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpKwogIHNjYWxlX3hfY29udGludW91cyhicmVha3M9c2VxKDE5NTAsMjAyMCxieT0xMCkpCmBgYAoKUGxvdCBwb3N0ZXJpb3IgcHJlZGljdGl2ZSBkaXN0cmlidXRpb24gYXQgZWFjaCB5ZWFyIHVudGlsIDIwMzAKYGFkZF9wcmVkaWN0ZWRfZHJhd3MoKWAgdGFrZXMgdGhlIHllYXJzIGZyb20gdGhlIGRhdGEgYW5kIHVzZXMKYGZpdF9saW5gIHRvIG1ha2UgdGhlIHByZWRpY3Rpb25zLgoKYGBge3J9CmRhdGFfbGluIHw+CiAgYWRkX3Jvdyh5ZWFyPTIwMjM6MjAzMCkgfD4KICBhZGRfcHJlZGljdGVkX2RyYXdzKGZpdF9saW4pIHw+CiAgIyBwbG90IGRhdGEKICBnZ3Bsb3QoYWVzKHg9eWVhciwgeT10ZW1wKSkgKwogIGdlb21fcG9pbnQoY29sb3I9MikgKwogICMgcGxvdCBsaW5lcmliYm9uIGZvciB0aGUgbGluZWFyIG1vZGVsCiAgc3RhdF9saW5lcmliYm9uKGFlcyh5ID0gLnByZWRpY3Rpb24pLCAud2lkdGggPSBjKC45NSksIGFscGhhID0gMS8yLCBjb2xvcj1icmV3ZXIucGFsKDUsICJCbHVlcyIpW1s1XV0pICsKICAjIGRlY29yYXRpb24KICBzY2FsZV9maWxsX2JyZXdlcigpKwogIGxhYnMoeD0gIlllYXIiLCB5ID0gJ1N1bW1lciB0ZW1wLiBAS2lscGlzasOkcnZpJykgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpKwogIHNjYWxlX3hfY29udGludW91cyhicmVha3M9c2VxKDE5NTAsMjAzMCxieT0xMCkpCmBgYAoKUG9zdGVyaW9yIHByZWRpY3RpdmUgY2hlY2sgd2l0aCBkZW5zaXR5IG92ZXJsYXlzIGV4YW1pbmVzIHRoZSB3aG9sZQp0ZW1wZXJhdHVyZSBkaXN0cmlidXRpb24KCmBgYHtyfQpwcF9jaGVjayhmaXRfbGluLCB0eXBlPSdkZW5zX292ZXJsYXknLCBuZHJhd3M9MjApCmBgYAoKTE9PLVBJVCBjaGVjayBpcyBnb29kIGZvciBjaGVja2luZyB3aGV0aGVyIHRoZSBub3JtYWwgZGlzdHJpYnV0aW9uCmlzIHdlbGwgZGVzY3JpYmluZyB0aGUgdmFyaWF0aW9uIGFzIGl0IGlzIGV4YW1pbmVzIHRoZSBjYWxpYnJhdGlvbgpvZiBMT08gcHJlZGljdGl2ZSBkaXN0cmlidXRpb25zIGNvbmRpdG9uYWxseSBvbiBlYWNoIHllYXIuIExPTy1QSVQKcGxvdHkgbG9va3MgZ29vZC4KCmBgYHtyfQpwcF9jaGVjayhmaXRfbGluLCB0eXBlPSdsb29fcGl0X3FxJywgbmRyYXdzPTQwMDApCmBgYAoKIyBMaW5lYXIgU3R1ZGVudCdzICR0JCBtb2RlbAoKVGhlIHRlbXBlcmF0dXJlcyB1c2VkIGluIHRoZSBhYm92ZSBhbmFseXNlcyBhcmUgYXZlcmFnZXMgb3ZlciB0aHJlZQptb250aHMsIHdoaWNoIG1ha2VzIGl0IG1vcmUgbGlrZWx5IHRoYXQgdGhleSBhcmUgbm9ybWFsbHkKZGlzdHJpYnV0ZWQsIGJ1dCB0aGVyZSBjYW4gYmUgZXh0cmVtZSBldmVudHMgaW4gdGhlIGZlYXRoZXIgYW5kIHdlCmNhbiBjaGVjayB3aGV0aGVyIG1vcmUgcm9idXN0IFN0dWRlbnQncyAkdCQgb2JzZXJ2YXRpb24gbW9kZWwgd291bGQKZ2l2ZSBkaWZmZXJlbnQgcmVzdWx0cyAoYWx0aG91Z2ggTE9PLVBJVCBjaGVjayBkaWQgYWxyZWFkeSBpbmRpY2F0ZQp0aGF0IHRoZSBub3JtYWwgd291bGQgYmUgZ29vZCkuCgoKYGBge3IgcmVzdWx0cz0naGlkZSd9CmZpdF9saW5fdCA8LSBicm0odGVtcCB+IHllYXIsIGRhdGEgPSBkYXRhX2xpbiwgZmFtaWx5ID0gc3R1ZGVudCgpLAogICAgICAgICAgICAgICAgIHByaW9yID0gcHJpb3Ioc3R1ZGVudF90KDMsIDAsIDAuMDMpLCBjbGFzcz0nYicpLAogICAgICAgICAgICAgICAgIHNlZWQgPSBTRUVELCByZWZyZXNoID0gMCkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuIFRoZSBiX3llYXIKcG9zdGVyaW9yIGxvb2tzIHNpbWlsYXIgYXMgYmVmb3JlIGFuZCB0aGUgcG9zdGVyaW9yIGZvciBkZWdyZWVzIG9mCmZyZWVkb20gYG51YCBoYXMgbW9zdCBvZiB0aGUgcG9zdGVyaW9yIG1hc3MgZm9yIHF1aXRlIGxhcmdlIHZhbHVlcwppbmRpY2F0aW5nIHRoZXJlIGlzIG5vIHN0cm9uZyBzdXBwb3J0IGZvciB0aGljayB0YWlsZWQgdmFyaWF0aW9uIGluCmF2ZXJhZ2Ugc3VtbWVyIHRlbXBlcmF0dXJlcy4KCmBgYHtyfQpmaXRfbGluX3QKYGBgCgojIFBhcmV0by1zbW9vdGhlZCBpbXBvcnRhbmNlLXNhbXBsaW5nIGxlYXZlLW9uZS1vdXQgY3Jvc3MtdmFsaWRhdGlvbiAoUFNJUy1MT08pCgpXZSBjYW4gdXNlIGxlYXZlLW9uZS1vdXQgY3Jvc3MtdmFsaWRhdGlvbiB0byBjb21wYXJlIHRoZSBleHBlY3RlZCBwcmVkaWN0aXZlIHBlcmZvcm1hbmNlLgoKTE9PIGNvbXBhcmlzb24gc2hvd3Mgbm9ybWFsIGFuZCBTdHVkZW50J3MgJHQkIG1vZGVsIGhhdmUgc2ltaWxhciBwZXJmb3JtYW5jZS4KCmBgYHtyfQpsb29fY29tcGFyZShsb28oZml0X2xpbiksIGxvbyhmaXRfbGluX3QpKQpgYGAKCiMgSGV0ZXJvc2tlZGFzdGljIGxpbmVhciBtb2RlbAoKSGV0ZXJvc2tlZGFzdGljaXR5IGFzc3VtZXMgdGhhdCB0aGUgdmFyaWF0aW9uIGFyb3VuZCB0aGUgbGluZWFyCm1lYW4gY2FuIGFsc28gdmFyeS4gV2UgY2FuIGFsbG93IHNpZ21hIHRvIGRlcGVuZCBvbiB5ZWFyLCB0b28uCkFsdGhvdWdoIHRoZSBhZGRpdGlvbmFsIGNvbXBvbmVudCBpcyB3cml0dGVuIGFzIGBzaWdtYSB+IHllYXJgLCB0aGUKbG9nIGxpbmsgZnVuY3Rpb24gaXMgdXNlZCBhbmQgdGhlIG1vZGVsIGlzIGZvciBsb2coc2lnbWEpLiBgYmYoKWAgYWxsb3dzCmxpc3Rpbmcgc2V2ZXJhbCBmb3JtdWxhcy4KCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X2xpbl9oIDwtIGJybShiZih0ZW1wIH4geWVhciwKICAgICAgICAgICAgICAgICAgICBzaWdtYSB+IHllYXIpLAogICAgICAgICAgICAgICAgIGRhdGEgPSBkYXRhX2xpbiwgZmFtaWx5ID0gZ2F1c3NpYW4oKSwKICAgICAgICAgICAgICAgICBwcmlvciA9IHByaW9yKHN0dWRlbnRfdCgzLCAwLCAwLjAzKSwgY2xhc3M9J2InKSwKICAgICAgICAgICAgICAgICBzZWVkID0gU0VFRCwgcmVmcmVzaCA9IDApCmBgYAoKQ2hlY2sgdGhlIHN1bW1hcnkgb2YgdGhlIHBvc3RlcmlvciBhbmQgaW5mZXJlbmNlIGRpYWdub3N0aWNzLiBUaGUgYl95ZWFyCnBvc3RlcmlvciBsb29rcyBzaW1pbGFyIGFzIGJlZm9yZS4gVGhlIHBvc3RlcmlvciBmb3Igc2lnbWFfeWVhcgpsb29rcyBsaWtlIGhhdmluZyBtb3NzdCBvZiB0aGUgbWEgZm9yIG5lZ2F0aXZlIHZhbHVlcywgaW5kaWNhdGluZwpkZWNyZWFzZSBpbiB0ZW1wZXJhdHVyZSB2YXJpYXRpb24gYXJvdW5kIHRoZSBtZWFuLgoKYGBge3J9CmZpdF9saW5faApgYGAKCkhpc3RvZ3JhbSBvZiBiX3llYXIgYW5kIGJfc2lnbWFfeWVhcgoKYGBge3J9CmFzX2RyYXdzX2RmKGZpdF9saW5faCkgfD4KICBtY21jX2FyZWFzKHBhcnM9YygnYl95ZWFyJywgJ2Jfc2lnbWFfeWVhcicpKQpgYGAKCkFzIGxvZyh4KSBpcyBhbG1vc3QgbGluZWFyIHdoZW4geCBpcyBjbG9zZSB0byB6ZXJvLCB3ZSBjYW4gc2VlIHRoYXQgdGhlCnNpZ21hIGlzIGRlY3JlYXNpbmcgYWJvdXQgMSUgcGVyIHllYXIgKDk1JSBpbnRlcnZhbCBmcm9tIDAlIHRvIDIlKS4KClBsb3QgcG9zdGVyaW9yIHByZWRpY3RpdmUgZGlzdHJpYnV0aW9uIGF0IGVhY2ggeWVhciB1bnRpbCAyMDMwCmBhZGRfcHJlZGljdGVkX2RyYXdzKClgIHRha2VzIHRoZSB5ZWFycyBmcm9tIHRoZSBkYXRhIGFuZCB1c2VzCmBmaXRfbGluX2hgIHRvIG1ha2UgdGhlIHByZWRpY3Rpb25zLgoKYGBge3J9CmRhdGFfbGluIHw+CiAgYWRkX3Jvdyh5ZWFyPTIwMjM6MjAzMCkgfD4KICBhZGRfcHJlZGljdGVkX2RyYXdzKGZpdF9saW5faCkgfD4KICAjIHBsb3QgZGF0YQogIGdncGxvdChhZXMoeD15ZWFyLCB5PXRlbXApKSArCiAgZ2VvbV9wb2ludChjb2xvcj0yKSArCiAgIyBwbG90IGxpbmVyaWJib24gZm9yIHRoZSBsaW5lYXIgbW9kZWwKICBzdGF0X2xpbmVyaWJib24oYWVzKHkgPSAucHJlZGljdGlvbiksIC53aWR0aCA9IGMoLjk1KSwgYWxwaGEgPSAxLzIsIGNvbG9yPWJyZXdlci5wYWwoNSwgIkJsdWVzIilbWzVdXSkgKwogICMgZGVjb3JhdGlvbgogIHNjYWxlX2ZpbGxfYnJld2VyKCkrCiAgbGFicyh4PSAiWWVhciIsIHkgPSAnU3VtbWVyIHRlbXAuIEBLaWxwaXNqw6RydmknKSArCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uPSJub25lIikrCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcz1zZXEoMTk1MCwyMDMwLGJ5PTEwKSkKYGBgCgpNYWtlIHByaW9yIHNlbnNpdGl2aXR5IGFuYWx5c2lzIGJ5IHBvd2VyLXNjYWxpbmcgYm90aCBwcmlvciBhbmQgbGlrZWxpaG9vZC4KCmBgYHtyfQpwb3dlcnNjYWxlX3NlbnNpdGl2aXR5KGZpdF9saW5faCkkc2Vuc2l0aXZpdHkgfD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtdXRhdGUoYWNyb3NzKHdoZXJlKGlzLmRvdWJsZSksICB+bnVtKC54LCBkaWdpdHM9MikpKQpgYGAKCldlIGNhbiB1c2UgbGVhdmUtb25lLW91dCBjcm9zcy12YWxpZGF0aW9uIHRvIGNvbXBhcmUgdGhlIGV4cGVjdGVkIHByZWRpY3RpdmUgcGVyZm9ybWFuY2UuCgpMT08gY29tcGFyaXNvbiBzaG93cyBob21vc2tlZGFzdGljIG5vcm1hbCBhbmQgaGV0ZXJvc2tlZGFzdGljCm5vcm1hbCBtb2RlbHMgaGF2ZSBzaW1pbGFyIHBlcmZvcm1hbmNlcy4KCmBgYHtyfQpsb29fY29tcGFyZShsb28oZml0X2xpbiksIGxvbyhmaXRfbGluX2gpKQpgYGAKCiMgSGV0ZXJvc2tlZGFzdGljIG5vbi1saW5lYXIgbW9kZWwKCldlIGNhbiB0ZXN0IHRoZSBsaW5lYXJpdHkgYXNzdW1wdGlvbiBieSB1c2luZyBub24tbGluZWFyIHNwbGluZQpmdW5jdGlvbnMsIGJ5IHVpbmcgYHMoeWVhcilgIHRlcm1zLiBTYW1wbGluZyBpcyBzbG93ZXIgYXMgdGhlCnBvc3RlcmlvciBnZXRzIG1vcmUgY29tcGxleC4KCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X3NwbGluZV9oIDwtIGJybShiZih0ZW1wIH4gcyh5ZWFyKSwKICAgICAgICAgICAgICAgICAgICAgc2lnbWEgfiBzKHllYXIpKSwKICAgICAgICAgICAgICAgICAgZGF0YSA9IGRhdGFfbGluLCBmYW1pbHkgPSBnYXVzc2lhbigpLAogICAgICAgICAgICAgICAgICBzZWVkID0gU0VFRCwgcmVmcmVzaCA9IDApCmBgYAoKV2UgZ2V0IHdhcm5pbmdzIGFib3V0IGRpdmVyZ2VuY2VzLCBhbmQgdHJ5IHJlcnVubmluZyB3aXRoIGhpZ2hlcgphZGFwdF9kZWx0YSwgd2hpY2ggbGVhZHMgdG8gdXNpbmcgc21hbGxlciBzdGVwIHNpemVzLiBPZnRlbgpgYWRhcHRfZGVsdGE9MC45OTlgIGxlYWRzIHRvIHZlcnkgc2xvdyBzYW1wbGluZywgYnV0IHdpdGggdGhpcwpzbWFsbCBkYXRhLCB0aGlzIGlzIG5vdCBhbiBpc3N1ZS4KCmBgYHtyfQpmaXRfc3BsaW5lX2ggPC0gdXBkYXRlKGZpdF9zcGxpbmVfaCwgY29udHJvbCA9IGxpc3QoYWRhcHRfZGVsdGE9MC45OTkpKQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4gV2UncmUgbm90CmFueW1vcmUgYWJsZSB0byBtYWtlIGludGVycHJldGF0aW9uIG9mIHRoZSB0ZW1wZXJhdHVyZSBpbmNyZWFzZQpiYXNlZCBvbiB0aGlzIHN1bW1hcnkuIEZvciBzcGxpbmVzLCB3ZSBzZWUgcHJpb3Igc2NhbGVzIGBzZHNgIGZvcgp0aGUgc3BsaW5lIGNvZWZmaWNpZW50cy4KCmBgYHtyfQpmaXRfc3BsaW5lX2gKYGBgCgpXZSBjYW4gc3RpbGwgcGxvdCBwb3N0ZXJpb3IgcHJlZGljdGl2ZSBkaXN0cmlidXRpb24gYXQgZWFjaCB5ZWFyCnVudGlsIDIwMzAgYGFkZF9wcmVkaWN0ZWRfZHJhd3MoKWAgdGFrZXMgdGhlIHllYXJzIGZyb20gdGhlIGRhdGEKYW5kIHVzZXMgYGZpdF9saW5faGAgdG8gbWFrZSB0aGUgcHJlZGljdGlvbnMuCgpgYGB7cn0KZGF0YV9saW4gfD4KICBhZGRfcm93KHllYXI9MjAyMzoyMDMwKSB8PgogIGFkZF9wcmVkaWN0ZWRfZHJhd3MoZml0X3NwbGluZV9oKSB8PgogICMgcGxvdCBkYXRhCiAgZ2dwbG90KGFlcyh4PXllYXIsIHk9dGVtcCkpICsKICBnZW9tX3BvaW50KGNvbG9yPTIpICsKICAjIHBsb3QgbGluZXJpYmJvbiBmb3IgdGhlIGxpbmVhciBtb2RlbAogIHN0YXRfbGluZXJpYmJvbihhZXMoeSA9IC5wcmVkaWN0aW9uKSwgLndpZHRoID0gYyguOTUpLCBhbHBoYSA9IDEvMiwgY29sb3I9YnJld2VyLnBhbCg1LCAiQmx1ZXMiKVtbNV1dKSArCiAgIyBkZWNvcmF0aW9uCiAgc2NhbGVfZmlsbF9icmV3ZXIoKSsKICBsYWJzKHg9ICJZZWFyIiwgeSA9ICdTdW1tZXIgdGVtcC4gQEtpbHBpc2rDpHJ2aScpICsKICB0aGVtZShsZWdlbmQucG9zaXRpb249Im5vbmUiKSsKICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzPXNlcSgxOTUwLDIwMzAsYnk9MTApKQpgYGAKCkFuZCB3ZSBjYW4gdXNlIGxlYXZlLW9uZS1vdXQgY3Jvc3MtdmFsaWRhdGlvbiB0byBjb21wYXJlIHRoZQpleHBlY3RlZCBwcmVkaWN0aXZlIHBlcmZvcm1hbmNlLgoKTE9PIGNvbXBhcmlzb24gc2hvd3MgaG9tb3NrZWRhc3RpYyBub3JtYWwgbGluZWFyIGFuZApoZXRlcm9za2VkYXN0aWMgbm9ybWFsIHNwbGluZSBtb2RlbHMgaGF2ZSBzaW1pbGFyCnBlcmZvcm1hbmNlcy4gVGhlcmUgYXJlIG5vdCBlbm91Z2ggb2JzZXJ2YXRpb25zIHRvIG1ha2UgY2xlYXIKZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBtb2RlbHMuCgpgYGB7cn0KbG9vX2NvbXBhcmUobG9vKGZpdF9saW4pLCBsb28oZml0X3NwbGluZV9oKSkKYGBgCgpGb3Igc3BsaW5lIGFuZCBvdGhlciBub24tcGFyYW1ldHJpYyBtb2RlbHMsIHdlIGNhbiB1c2UgcHJlZGljdGl2ZQplc3RpbWF0ZXMgYW5kIHByZWRpY3Rpb25zIHRvIGdldCBpbnRlcnByZXRhYmxlIHF1YW50aXRpZXMuIExldCdzCmV4YW1pbmUgdGhlIGRpZmZlcmVuY2Ugb2YgZXN0aW1hdGVkIGF2ZXJhZ2UgdGVtcGVyYXR1cmUgaW4geWVhcnMKMTk1MiBhbmQgMjAyMi4KCmBgYHtyfQp0ZW1wX2RpZmYgPC0gcG9zdGVyaW9yX2VwcmVkKGZpdF9zcGxpbmVfaCwgbmV3ZGF0YT1maWx0ZXIoZGF0YV9saW4seWVhcj09MTk1Mnx5ZWFyPT0yMDIyKSkgfD4KICBydmFyKCkgfD4KICBkaWZmKCkgfD4KICBhc19kcmF3c19kZigpIHw+CiAgc2V0X3ZhcmlhYmxlcygndGVtcF9kaWZmJykKCnRlbXBfZGlmZiA8LSBkYXRhX2xpbiB8PgogIGZpbHRlcih5ZWFyPT0xOTUyfHllYXI9PTIwMjIpIHw+CiAgYWRkX2VwcmVkX2RyYXdzKGZpdF9zcGxpbmVfaCkgfD4KICBwaXZvdF93aWRlcihpZF9jb2xzPS5kcmF3LCBuYW1lc19mcm9tID0geWVhciwgdmFsdWVzX2Zyb20gPSAuZXByZWQpIHw+CiAgbXV0YXRlKHRlbXBfZGlmZiA9IGAyMDIyYC1gMTk1MmAsCiAgICAgICAgIC5jaGFpbiA9ICguZHJhdyAtIDEpICUvJSAxMDAwICsgMSwKICAgICAgICAgLml0ZXJhdGlvbiA9ICguZHJhdyAtIDEpICUlIDEwMDAgKyAxKSB8PgogIGFzX2RyYXdzX2RmKCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J3RlbXBfZGlmZicpCmBgYAoKUG9zdGVyaW9yIGRpc3RyaWJ1dGlvbiBmb3IgYXZlcmFnZSBzdW1tZXIgdGVtcGVyYXR1cmUgaW5jcmVhc2UgZnJvbSAxOTUyIHRvIDIwMjIKCmBgYHtyfQp0ZW1wX2RpZmYgfD4KICBtY21jX2hpc3QoKQpgYGAKCjk1JSBwb3N0ZXJpb3IgaW50ZXJ2YWwgZm9yIGF2ZXJhZ2Ugc3VtbWVyIHRlbXBlcmF0dXJlIGluY3JlYXNlIGZyb20gMTk1MiB0byAyMDIyCgpgYGB7cn0KdGVtcF9kaWZmIHw+CiAgc3VtbWFyaXNlX2RyYXdzKH5xdWFudGlsZSgueCwgcHJvYnMgPSBjKDAuMDI1LCAwLjk3NSkpLAogICAgICAgICAgICAgICAgICB+bWNzZV9xdWFudGlsZSgueCwgcHJvYnMgPSBjKDAuMDI1LCAwLjk3NSkpLAogICAgICAgICAgICAgICAgICAubnVtX2FyZ3MgPSBsaXN0KGRpZ2l0cyA9IDIsIG5vdGF0aW9uID0gImRlYyIpKQpgYGAKCk1ha2UgcHJpb3Igc2Vuc2l0aXZpdHkgYW5hbHlzaXMgYnkgcG93ZXItc2NhbGluZyBib3RoIHByaW9yIGFuZApsaWtlbGlob29kIHdpdGggZm9jdXMgb24gYXZlcmFnZSBzdW1tZXIgdGVtcGVyYXR1cmUgaW5jcmVhc2UgZnJvbQoxOTUyIHRvIDIwMjIuCgpgYGB7cn0KcG93ZXJzY2FsZV9zZW5zaXRpdml0eShmaXRfc3BsaW5lX2gsIHByZWRpY3Rpb24gPSBcKHgsIC4uLikgdGVtcF9kaWZmLCBudW1fYXJncz1saXN0KGRpZ2l0cz0yKQogICAgICAgICAgICAgICAgICAgICAgICkkc2Vuc2l0aXZpdHkgfD4KICAgICAgICAgICAgICAgICAgICAgICAgIGZpbHRlcih2YXJpYWJsZT09J3RlbXBfZGlmZicpIHw+CiAgICAgICAgICAgICAgICAgICAgICAgICBtdXRhdGUoYWNyb3NzKHdoZXJlKGlzLmRvdWJsZSksICB+bnVtKC54LCBkaWdpdHM9MikpKQpgYGAKClByb2JhYmlsaXR5IHRoYXQgdGhlIGF2ZXJhZ2Ugc3VtbWVyIHRlbXBlcmF0dXJlIGhhcyBpbmNyZWFzZWQgZnJvbQoxOTUyIHRvIDIwMjIgaXMgOTkuNSUuCgpgYGB7cn0KdGVtcF9kaWZmIHw+CiAgbXV0YXRlKElfdGVtcF9kaWZmX2d0XzAgPSB0ZW1wX2RpZmY+MCwKICAgICAgICAgdGVtcF9kaWZmID0gTlVMTCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J0lfdGVtcF9kaWZmX2d0XzAnKSB8PgogIHN1bW1hcmlzZV9kcmF3cyhtZWFuLCBtY3NlX21lYW4pCmBgYAoKCiMgQ29tcGFyaXNvbiBvZiBrIGdyb3VwcyB3aXRoIGhpZXJhcmNoaWNhbCBub3JtYWwgbW9kZWxzCgpMb2FkIGZhY3RvcnkgZGF0YSwgd2hpY2ggY29udGFpbiA1IHF1YWxpdHkgbWVhc3VyZW1lbnRzIGZvciBlYWNoIG9mCjYgbWFjaGluZXMuIFdlJ3JlIGludGVyZXN0ZWQgaW4gYW5hbHlzaW5nIGFyZSB0aGUgcXVhbGl0eSBkaWZmZXJlbmNlcwpiZXR3ZWVuIHRoZSBtYWNoaW5lcy4KCmBgYHtyfQpmYWN0b3J5IDwtIHJlYWQudGFibGUodXJsKCdodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vYXZlaHRhcmkvQkRBX2NvdXJzZV9BYWx0by9tYXN0ZXIvcnBhY2thZ2UvZGF0YS1yYXcvZmFjdG9yeS50eHQnKSkKY29sbmFtZXMoZmFjdG9yeSkgPC0gMTo2CmZhY3RvcnkKYGBgCgpXZSBwaXZvdCB0aGUgZGF0YSB0byBsb25nIGZvcm1hdAoKYGBge3J9CmZhY3RvcnkgPC0gZmFjdG9yeSB8PgogIHBpdm90X2xvbmdlcihjb2xzID0gZXZlcnl0aGluZygpLAogICAgICAgICAgICAgICBuYW1lc190byA9ICdtYWNoaW5lJywKICAgICAgICAgICAgICAgdmFsdWVzX3RvID0gJ3F1YWxpdHknKQpmYWN0b3J5CmBgYAoKIyMgUG9vbGVkIG1vZGVsCgpBcyBjb21wYXJpc29uIG1ha2UgYWxzbyBwb29sZWQgbW9kZWwKCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfcG9vbGVkIDwtIGJybShxdWFsaXR5IH4gMSwgZGF0YSA9IGZhY3RvcnksIHJlZnJlc2g9MCkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuCgpgYGB7cn0KZml0X3Bvb2xlZApgYGAKCiMjIFNlcGFyYXRlIG1vZGVsCgpBcyBjb21wYXJpc29uIG1ha2UgYWxzbyBzZXByYXRlIG1vZGVsLiBUbyBtYWtlIGl0IGNvbXBsZXRlbHkKc2VwYXJhdGUgd2UgbmVlZCB0byBoYXZlIGRpZmZlcmVudCBzaWdtYSBmb3IgZWFjaCBtYWNoaW5lLCB0b28uCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X3NlcGFyYXRlIDwtIGJybShiZihxdWFsaXR5IH4gMCArIG1hY2hpbmUsCiAgICAgICAgICAgICAgICAgICAgICAgc2lnbWEgfiAwICsgbWFjaGluZSksCiAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGZhY3RvcnksIHJlZnJlc2g9MCkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuCgpgYGB7cn0KZml0X3NlcGFyYXRlCmBgYAoKIyBDb21tb24gdmFyaWFuY2UgaGllcmFyY2hpY2FsIG1vZGVsIChBTk9WQSkKCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfaGllciA8LSBicm0ocXVhbGl0eSB+IDEgKyAoMSB8IG1hY2hpbmUpLAogICAgICAgICAgICAgICAgZGF0YSA9IGZhY3RvcnksIHJlZnJlc2ggPSAwKQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4KCmBgYHtyfQpmaXRfaGllcgpgYGAKCkxPTyBjb21wYXJpc29uIHNob3dzIHRoZSBoaWVyYXJjaGljYWwgbW9kZWwgaXMgdGhlIGJlc3QuIFRoZQpkaWZmZXJlbmNlcyBhcmUgc21hbGwgYXMgdGhlIG51bWJlciBvZiBvYnNlcnZhdGlvbnMgaXMgc21hbGwgYW5kCnRoZXJlIGlzIGEgY29uc2lkZXJhYmxlIHByZWRpY3Rpb24gKGFsZWF0b3JpYykgdW5jZXJ0YWludHkuCgpgYGB7cn0KbG9vX2NvbXBhcmUobG9vKGZpdF9wb29sZWQpLCBsb28oZml0X3NlcGFyYXRlKSwgbG9vKGZpdF9oaWVyKSkKYGBgCgpEaWZmZXJlbnQgbW9kZWwgcG9zdGVyaW9yIGRpc3RyaWJ1dGlvbnMgZm9yIHRoZSBtZWFuCnF1YWxpdHkuIFBvb2xlZCBtb2RlbCBpZ25vcmVzIHRoZSB2YXJpdGlvbiBiZXR3ZWVuCm1hY2hpbmVzLiBTZXBhcmF0ZSBtb2RlbCBkb2Vzbid0IHRha2UgYmVuZWZpdCBmcm9tIHRoZSBzaW1pbGFyaXkgb2YKdGhlIG1hY2hpbmVzIGFuZCBoYXMgaGlnaGVyIHVuY2VydGFpbnR5LgoKYGBge3J9CnBoIDwtIGZpdF9oaWVyIHw+CiAgc3ByZWFkX3J2YXJzKGJfSW50ZXJjZXB0LCByX21hY2hpbmVbbWFjaGluZSxdKSB8PgogIG11dGF0ZShtYWNoaW5lX21lYW4gPSBiX0ludGVyY2VwdCArIHJfbWFjaGluZSkgfD4KICBnZ3Bsb3QoYWVzKHhkaXN0PW1hY2hpbmVfbWVhbiwgeT1tYWNoaW5lKSkgKwogIHN0YXRfaGFsZmV5ZSgpICsKICBzY2FsZV95X2NvbnRpbnVvdXMoYnJlYWtzPTE6NikgKwogIGxhYnMoeD0nUXVhbGl0eScsIHk9J01hY2hpbmUnLCB0aXRsZT0nSGllcmFyY2hpY2FsJykKCnBzIDwtIGZpdF9zZXBhcmF0ZSB8PgogIGFzX2RyYXdzX2RmKCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J2JfbWFjaGluZScsIHJlZ2V4PVRSVUUpIHw+CiAgc2V0X3ZhcmlhYmxlcyhwYXN0ZTAoJ2JfbWFjaGluZVsnLCAxOjYsICddJykpIHw+CiAgYXNfZHJhd3NfcnZhcnMoKSB8PgogIHNwcmVhZF9ydmFycyhiX21hY2hpbmVbbWFjaGluZV0pIHw+CiAgbXV0YXRlKG1hY2hpbmVfbWVhbiA9IGJfbWFjaGluZSkgfD4KICBnZ3Bsb3QoYWVzKHhkaXN0PW1hY2hpbmVfbWVhbiwgeT1tYWNoaW5lKSkgKwogIHN0YXRfaGFsZmV5ZSgpICsKICBzY2FsZV95X2NvbnRpbnVvdXMoYnJlYWtzPTE6NikgKwogIGxhYnMoeD0nUXVhbGl0eScsIHk9J01hY2hpbmUnLCB0aXRsZT0nU2VwYXJhdGUnKQoKcHAgPC0gZml0X3Bvb2xlZCB8PgogIHNwcmVhZF9ydmFycyhiX0ludGVyY2VwdCkgfD4KICBtdXRhdGUobWFjaGluZV9tZWFuID0gYl9JbnRlcmNlcHQpIHw+CiAgZ2dwbG90KGFlcyh4ZGlzdD1tYWNoaW5lX21lYW4sIHk9MCkpICsKICBzdGF0X2hhbGZleWUoKSArCiAgc2NhbGVfeV9jb250aW51b3VzKGJyZWFrcz1OVUxMKSArCiAgbGFicyh4PSdRdWFsaXR5JywgeT0nQWxsIG1hY2hpbmVzJywgdGl0bGU9J1Bvb2xlZCcpCgoocHAgLyBwcyAvIHBoKSAqIHhsaW0oYyg1MCwxNDApKQpgYGAKCk1ha2UgcHJpb3Igc2Vuc2l0aXZpdHkgYW5hbHlzaXMgYnkgcG93ZXItc2NhbGluZyBib3RoIHByaW9yIGFuZApsaWtlbGlob29kIHdpdGggZm9jdXMgb24gbWVhbiBxdWFsaXR5IG9mIGVhY2ggbWFjaGluZS4gV2Ugc2VlIG5vCnByaW9yIHNlbnNpdGl2aXR5LgoKYGBge3J9Cm1hY2hpbmVfbWVhbiA8LSBmaXRfaGllciB8PgogIGFzX2RyYXdzX2RmKCkgfD4KICBtdXRhdGUoYWNyb3NzKG1hdGNoZXMoJ3JfbWFjaGluZScpLCB+IC54IC0gYl9JbnRlcmNlcHQpKSB8PgogIHN1YnNldF9kcmF3cyh2YXJpYWJsZT0ncl9tYWNoaW5lJywgcmVnZXg9VFJVRSkgfD4KICBzZXRfdmFyaWFibGVzKHBhc3RlMCgnbWFjaGluZV9tZWFuWycsIDE6NiwgJ10nKSkKcG93ZXJzY2FsZV9zZW5zaXRpdml0eShmaXRfaGllciwgcHJlZGljdGlvbiA9IFwoeCwgLi4uKSBtYWNoaW5lX21lYW4sIG51bV9hcmdzPWxpc3QoZGlnaXRzPTIpCiAgICAgICAgICAgICAgICAgICAgICAgKSRzZW5zaXRpdml0eSB8PgogICAgICAgICAgICAgICAgICAgICAgICAgZmlsdGVyKHN0cl9kZXRlY3QodmFyaWFibGUsJ21hY2hpbmVfbWVhbicpKSB8PgogICAgICAgICAgICAgICAgICAgICAgICAgbXV0YXRlKGFjcm9zcyh3aGVyZShpcy5kb3VibGUpLCAgfm51bSgueCwgZGlnaXRzPTIpKSkKYGBgCgoKIyBIaWVyYXJjaGljYWwgYmlub21pYWwgbW9kZWwKCltTb3JhZmVuaWIgVG94aWNpdHkgRGF0YXNldCBpbiBgbWV0YWRhdGAgUiBwYWNrYWdlXShodHRwczovL3d2aWVjaHRiLmdpdGh1Yi5pby9tZXRhZGF0L3JlZmVyZW5jZS9kYXQudXJzaW5vMjAyMS5odG1sKQppbmNsdWRlcyByZXN1bHRzIGZyb20gMTMgc3R1ZGllcyBpbnZlc3RpZ2F0aW5nIHRoZSBvY2N1cnJlbmNlIG9mCmRvc2UgbGltaXRpbmcgdG94aWNpdGllcyAoRExUcykgYXQgZGlmZmVyZW50IGRvc2VzIG9mIFNvcmFmZW5pYi4KCkxvYWQgZGF0YQoKYGBge3J9CmxvYWQodXJsKCdodHRwczovL2dpdGh1Yi5jb20vd3ZpZWNodGIvbWV0YWRhdC9yYXcvbWFzdGVyL2RhdGEvZGF0LnVyc2lubzIwMjEucmRhJykpCmhlYWQoZGF0LnVyc2lubzIwMjEpCmBgYAoKTnVtYmVyIG9mIHBhdGllbnRzIHBlciBzdHVkeQoKYGBge3J9CmRhdC51cnNpbm8yMDIxIHw+CiAgZ3JvdXBfYnkoc3R1ZHkpIHw+CiAgc3VtbWFyaXNlKE4gPSBzdW0odG90YWwpKSB8PgogIGdncGxvdChhZXMoeD1OLCB5PXN0dWR5KSkgKwogIGdlb21fY29sKGZpbGw9NCkgKwogIGxhYnMoeD0nTnVtYmVyIG9mIHBhdGllbnRzIHBlciBzdHVkeScsIHk9J1N0dWR5JykKYGBgCgpEaXN0cmlidXRpb24gb2YgZG9zZXMKCmBgYHtyfQpkYXQudXJzaW5vMjAyMSB8PgogIGdncGxvdChhZXMoeD1kb3NlKSkgKwogIGdlb21faGlzdG9ncmFtKGJyZWFrcz1zZXEoNTAsMTA1MCxieT0xMDApLCBmaWxsPTQsIGNvbG91cj0xKSArCiAgbGFicyh4PSdEb3NlIChtZyknLCB5PSdDb3VudCcpICsKICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzPXNlcSgxMDAsMTAwMCxieT0xMDApKQpgYGAKCkVhY2ggc3R1ZHkgaXMgdXNpbmcgJDItLTYkIGRpZmZlcmVudCBkb3NlIGxldmVscy4gVGhyZWUgc3R1ZGllcwp0aGF0IGluY2x1ZGUgb25seSB0d28gZG9zZSBsZXZlbHMgYXJlIGxpa2VsbHkgdG8gcHJvdmlkZSB3ZWFrCmluZm9ybWF0aW9uIG9uIHNsb3BlLgoKYGBge3J9CmNyb3NzdGFiIDwtIHdpdGgoZGF0LnVyc2lubzIwMjEsdGFibGUoZG9zZSxzdHVkeSkpCmRhdGEuZnJhbWUoY291bnQ9Y29sU3Vtcyhjcm9zc3RhYiksIHN0dWR5PWNvbG5hbWVzKGNyb3NzdGFiKSkgfD4KICBnZ3Bsb3QoYWVzKHg9Y291bnQsIHk9c3R1ZHkpKSArCiAgZ2VvbV9jb2woZmlsbD00KSArCiAgbGFicyh4PSdOdW1iZXIgb2YgZG9zZSBsZXZlbHMgcGVyIHN0dWR5JywgeT0nU3R1ZHknKQpgYGAKClBvb2xlZCBtb2RlbCBhc3N1bWVzIGFsbCBzdHVkaWVzIGhhdmUgdGhlIHNhbWUgZG9zZSBlZmZlY3QKKHJlbWluZGVyOiBgfiBkb3NlYCBpcyBlcXVpdmFsZW50IHRvIGB+IDEgKyBkb3NlYCkKCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfcG9vbGVkIDwtIGJybShldmVudHMgfCB0cmlhbHModG90YWwpIH4gZG9zZSwKICAgICAgICAgICAgICAgICAgcHJpb3IgPSBjKHByaW9yKHN0dWRlbnRfdCg3LCAwLCAxLjUpLCBjbGFzcz0nSW50ZXJjZXB0JyksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmlvcihub3JtYWwoMCwgMSksIGNsYXNzPSdiJykpLAogICAgICAgICAgICAgICAgICBmYW1pbHk9Ymlub21pYWwoKSwgZGF0YT1kYXQudXJzaW5vMjAyMSkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuCgpgYGB7cn0KZml0X3Bvb2xlZApgYGAKCkRvc2UgY29lZmZpY2llbnQgc2VlbXMgdG8gYmUgdmVyeSBzbWFsbC4gTG9va2luZyBhdCB0aGUgcG9zdGVyaW9yLAp3ZSBzZWUgdGhhdCBpdCBpcyBwb3NpdGl2ZSB3aXRoIGhpZ2ggcHJvYmFiaWxpdHkuIAoKYGBge3J9CmZpdF9wb29sZWQgfD4KICBhc19kcmF3cygpIHw+CiAgc3Vic2V0X2RyYXdzKHZhcmlhYmxlPSdiX2Rvc2UnKSB8PgogIHN1bW1hcmlzZV9kcmF3cyh+cXVhbnRpbGUoLngsIHByb2JzID0gYygwLjAyNSwgMC45NzUpKSwgfm1jc2VfcXVhbnRpbGUoLngsIHByb2JzID0gYygwLjAyNSwgMC45NzUpKSkKYGBgCgpUaGUgZG9zZSB3YXMgcmVwb3J0ZWQgaW4gbWcsIGFuZCBtb3N0IHZhbHVlcyBhcmUgaW4gaHVuZHJlZHMuIEl0IGlzCm9mdGVuIHNlbnNpYmxlIHRvIHN3aXRjaCB0byBhIHNjYWxlIGluIHdoaWNoIHRoZSByYW5nZSBvZiB2YWx1ZXMgaXMKY2xvc2VyIHRvIHVuaXQgcmFuZ2UuIEluIHRoaXMgY2FzZSBpdCBpcyBuYXR1cmFsIHRvIHVzZSBnIGluc3RlYWQKb2YgbWcuCgpgYGB7cn0KZGF0LnVyc2lubzIwMjEgPC0gZGF0LnVyc2lubzIwMjEgfD4KICBtdXRhdGUoZG9zZWcgPSBkb3NlLzEwMDApCmBgYAoKRml0IHRoZSBwb29sZWQgbW9kZWwgYWdhaW4gdWluZyBgZG9zZWdgCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X3Bvb2xlZCA8LSBicm0oZXZlbnRzIHwgdHJpYWxzKHRvdGFsKSB+IGRvc2VnLAogICAgICAgICAgICAgICAgICBwcmlvciA9IGMocHJpb3Ioc3R1ZGVudF90KDcsIDAsIDEuNSksIGNsYXNzPSdJbnRlcmNlcHQnKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByaW9yKG5vcm1hbCgwLCAxKSwgY2xhc3M9J2InKSksCiAgICAgICAgICAgICAgICAgIGZhbWlseT1iaW5vbWlhbCgpLCBkYXRhPWRhdC51cnNpbm8yMDIxKQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4KCmBgYHtyfQpmaXRfcG9vbGVkCmBgYAoKTm93IGl0IGlzIGVhc2llciB0byBpbnRlcnByZXQgdGhlIHByZXNlbnRlZCB2YWx1ZXMuIApTZXBhcmF0ZSBtb2RlbCBhc3N1bWVzIGFsbCBzdHVkaWVzIGhhdmUgZGlmZmVyZW50IGRvc2UgZWZmZWN0LgpJdCB3b3VsZCBiZSBhIGJpdCBjb21wbGljYXRlZCB0byBzZXQgYSBkaWZmZXJlbnQgcHJpb3Igb24gc3R1ZHkgc3BlY2lmaWMKaW50ZXJjZXB0cyBhbmQgb3RoZXIgY29lZmZpY2llbnRzLCBzbyB3ZSB1c2UgdGhlIHNhbWUgcHJpb3IgZm9yIGFsbC4KCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfc2VwYXJhdGUgPC0gYnJtKGV2ZW50cyB8IHRyaWFscyh0b3RhbCkgfiAwICsgc3R1ZHkgKyBkb3NlZzpzdHVkeSwKICAgICAgICAgICAgICAgICAgICBwcmlvcj1wcmlvcihzdHVkZW50X3QoNywgMCwgMS41KSwgY2xhc3M9J2InKSwKICAgICAgICAgICAgICAgICAgICBmYW1pbHk9Ymlub21pYWwoKSwgZGF0YT1kYXQudXJzaW5vMjAyMSkKYGBgCgpDaGVjayB0aGUgc3VtbWFyeSBvZiB0aGUgcG9zdGVyaW9yIGFuZCBpbmZlcmVuY2UgZGlhZ25vc3RpY3MuCgpgYGB7cn0KZml0X3NlcGFyYXRlCmBgYAoKV2UgYnVpbGQgdHdvIGRpZmZlcmVudCBoaWVyYXJjaGljYWwgbW9kZWxzLiBUaGUgZmlyc3Qgb25lIGhhcwpoaWVyYXJjaGljYWwgbW9kZWwgZm9yIHRoZSBpbnRlcmNlcHQsIHRoYXQgaXMsIGVhY2ggc3R1ZHkgaGFzIGEKcGFyYW1ldGVyIHRlbGxpbmcgaG93IG11Y2ggdGhhdCBzdHVkeSBkaWZmZXJzIGZyb20gdGhlIGNvbW1vbgpwb3B1bGF0aW9uIGludGVyY2VwdC4KCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfaGllcjEgPC0gYnJtKGV2ZW50cyB8IHRyaWFscyh0b3RhbCkgfiBkb3NlZyArICgxIHwgc3R1ZHkpLAogICAgICAgICAgICAgICAgICAgIHByaW9yPWMocHJpb3Ioc3R1ZGVudF90KDcsIDAsIDEuNSksIGNsYXNzPSdJbnRlcmNlcHQnKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByaW9yKG5vcm1hbCgwLCAxKSwgY2xhc3M9J2InKSksCiAgICAgICAgICAgICAgICBmYW1pbHk9Ymlub21pYWwoKSwgZGF0YT1kYXQudXJzaW5vMjAyMSkKYGBgCgpUaGUgc2Vjb25kIGhpZXJhcmNoaWNhbCBtb2RlbCBhc3N1bWVzIHRoYXQgYWxzbyB0aGUgc2xvcGUgY2FuIHZhcnkKYmV0d2VlbiB0aGUgc3R1ZGllcy4KCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfaGllcjIgPC0gYnJtKGV2ZW50cyB8IHRyaWFscyh0b3RhbCkgfiBkb3NlZyArIChkb3NlZyB8IHN0dWR5KSwKICAgICAgICAgICAgICAgICAgICBwcmlvcj1jKHByaW9yKHN0dWRlbnRfdCg3LCAwLCAxLjUpLCBjbGFzcz0nSW50ZXJjZXB0JyksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmlvcihub3JtYWwoMCwgMSksIGNsYXNzPSdiJykpLAogICAgICAgICAgICAgICAgZmFtaWx5PWJpbm9taWFsKCksIGRhdGE9ZGF0LnVyc2lubzIwMjEpCmBgYAoKV2Ugc2VlbSBzb21lIGRpdmVyZ2VuY2VzIGR1ZSB0byBoaWdobHkgdmFyeWluZyBwb3N0ZXJpb3IKY3VydmF0dXJlLiBXZSByZXBlYXQgdGhlIHNhbXBsaW5nIHdpdGggaGlnaGVyIGFkYXB0X2RlbHRhLCB3aGljaAphZGp1c3QgdGhlIHN0ZXAgc2l6ZSB0byBiZSBzbWFsbGVyLiBIaWdoZXIgYWRhcHRfZGVsdGEgbWFrZXMgdGhlCmNvbXB1dGF0aW9uIHNsb3dlciwgYnV0IHRoYXQgaXMgbm90IGFuIGlzc3VlIGluIHRoaXMgY2FzZS4gSWYgeW91CmdldCBkaXZlcmdlbmNlcyB3aXRoIGBhZGFwdF9kZWx0YT0wLjk5YCwgaXQgaXMgbGlrZWx5IHRoYXQgZXZlbgpsYXJnZXIgdmFsdWVzIGRvbid0IGhlbHAsIGFuZCB5b3UgbmVlZCB0byBjb25zaWRlciBkaWZmZXJlbnQKcGFyYW1ldGVyaXNhdGlvbiwgZGlmZmVyZW50IG1vZGVsLCBvciBtb3JlIGluZm9ybWF0aXZlIHByaW9ycy4KCmBgYHtyIHJlc3VsdHM9J2hpZGUnfQpmaXRfaGllcjIgPC0gdXBkYXRlKGZpdF9oaWVyMiwgY29udHJvbD1saXN0KGFkYXB0X2RlbHRhPTAuOTkpKQpgYGAKCkxPTy1DViBjb21wYXJpc29uCgpgYGB7cn0KbG9vX2NvbXBhcmUobG9vKGZpdF9wb29sZWQpLCBsb28oZml0X3NlcGFyYXRlKSwgbG9vKGZpdF9oaWVyMSksIGxvbyhmaXRfaGllcjIpKQpgYGAKCldlIGdldCB3YXJuaW5ncyBhYm91dCBzZXZlcmFsIFBhcmV0byBrJ3MgPiAwLjcgaW4gUFNJUy1MT08gZm9yCnNlcGFyYXRlIG1vZGVsLCBidXQgYXMgaW4gdGhhdCBjYXNlIHRoZSBMT08tQ1YgZXN0aW1hdGUgaXMgdXN1YWxseQpvdmVyb3B0aW1pc3RpYyBhbmQgdGhlIHNlcGFyYXRlIG1vZGVsIGlzIHRoZSB3b3JzdCwgdGhlcmUgaXMgbm8KbmVlZCB0byB1c2UgbW9yZSBhY2N1cmF0ZSBjb21wdXRhdGlvbiBmb3IgdGhlIHNlcGFyYXRlIG1vZGVsLgoKV2UgZ2V0IHdhcm5pbmdzIGFib3V0IGEgZmV3IFBhcmV0byBrJ3MgPiAwLjcgaW4gUFNJUy1MT08gZm9yIGJvdGgKaGllcmFyY2hpY2FsIG1vZGVscy4gV2UgY2FuIGltcHJvdmUgdGhlIGFjY3VyYWN5IGJlIHJ1bm5pbmcgTUNNQwpmb3IgdGhlc2UgTE9PIGZvbGRzLiBXZSB1c2UgYGFkZF9jcml0ZXJpb24oKWAgZnVuY3Rpb24gdG8gc3RvcmUgdGhlCkxPTyBjb21wdXRhdGlvbiByZXN1bHRzIGFzIHRoZXkgdGFrZSBhIGJpdCBsb25nZXIgbm93LiBXZSBnZXQgc29tZQpkaXZlcmdlbmNlcyBpbiBjYXNlIG9mIHRoZSBzZWNvbmQgaGllcmFyY2hpY2FsIG1vZGVsLCBhcyBsZWF2aW5nCm91dCBhbiBvYnNlcnZhdGlvbiBmb3IgYSBzdHVkeSB0aGF0IGhhcyBvbmx5IHR3byBkb3NlIGxldmVscyBpcwptYWtpbmcgdGhlIHBvc3RlcmlvciBoYXZpbmcgYSBkaWZmaWN1bHQgc2hhcGUuCgpgYGB7cn0KZml0X2hpZXIxIDwtIGFkZF9jcml0ZXJpb24oZml0X2hpZXIxLCBjcml0ZXJpb249J2xvbycsIHJlbG9vPVRSVUUpCmZpdF9oaWVyMiA8LSBhZGRfY3JpdGVyaW9uKGZpdF9oaWVyMiwgY3JpdGVyaW9uPSdsb28nLCByZWxvbz1UUlVFKQpgYGAKCldlIHJlcGVhdCB0aGUgTE9PLUNWIGNvbXBhcmlzb24gKHdpdGhvdXQgc2VwYXJhdGUgbW9kZWwpLiBgbG9vKClgCmZ1bmN0aW9uIGlzIHVzZWluZyB0aGUgcmV1bHRzIGFkZGVkIHRvIHRoZSBmaXQgb2JqZWN0cy4KCmBgYHtyfQpsb29fY29tcGFyZShsb28oZml0X3Bvb2xlZCksIGxvbyhmaXRfaGllcjEpLCBsb28oZml0X2hpZXIyKSkKYGBgCgpUaGUgcmVzdWx0cyBkaWQgbm90IGNoYW5nZSBtdWNoLiBUaGUgZmlyc3QgaGllcmFyY2hpY2FsIG1vZGVsIGlzCnNsaWdodGx5IGJldHRlciB0aGFuIG90aGVyIG1vZGVscywgYnV0IGZvciBwcmVkaWN0aXZlIHB1cnBvc2VzCnRoZXJlIGlzIG5vdCBtdWNoIGRpZmZlcmVuY2UgKHRoZXJlIGlzIGhpZ2ggYWxlYXRvcmljIHVuY2VydGFpbnR5CmluIHRoZSBwcmVkaWN0aW9ucykuIEFkZGluZyBoaWVhcmNoaWNhbCBtb2RlbCBmb3IgdGhlIHNsb3BlLApkZWNyYXNlZCB0aGUgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBhbmQgdGh1cyBpdCBpcyBsaWtlbHkgdGhhdAp0aGVyZSBpcyBub3QgZW5vdWdoIGluZm9ybWF0aW9uIGFib3V0IHRoZSB2YXJpYXRpb24gaW4gc2xvcGVzCmJldHdlZW4gc3R1ZGllcy4KClBvc3RlcmlvciBwcmVkaWN0aXZlIGNoZWNraW5nIHNob3dpbmcgdGhlIG9ic2VydmVkIGFuZCBwcmVkaWN0ZWQKbnVtYmVyIG9mIGV2ZW50cy4gUm9vdGdyYW0gdXNlcyBzcXVhcmUgcm9vdCBvZiBjb3VudHMgb24geS1heGlzIGZvcgpiZXR0ZXIgc2NhbGluZy4gUm9vdG9ncmFtIGlzIHVzZWZ1bCBmb3IgY291bnQgZGF0YSB3aGVuIHRoZSByYW5nZQpvZiBjb3VudHMgaXMgc21hbGwgb3IgbW9kZXJhdGUuCgpgYGB7cn0KcHBfY2hlY2soZml0X3Bvb2xlZCwgdHlwZSA9ICJyb290b2dyYW0iKSArCiAgbGFicyh0aXRsZT0nUG9vbGVkIG1vZGVsJykKcHBfY2hlY2soZml0X2hpZXIxLCB0eXBlID0gInJvb3RvZ3JhbSIpICsKICBsYWJzKHRpdGxlPSdIaWVyYXJjaGljYWwgbW9kZWwnKQpwcF9jaGVjayhmaXRfaGllcjIsIHR5cGUgPSAicm9vdG9ncmFtIikgKwogIGxhYnModGl0bGU9J0hpZXJhcmNoaWNhbCBtb2RlbCcpCmBgYAoKV2Ugc2VlIHRoYXQgdGhlIGhpZXJhcmNoaWNhbCBtb2RlbHMgaGF2ZSBoaWdoZXIgcHJvYmFiaWxpdHkgZm9yCmZ1dHVyZSBjb3VudHMgdGhhdCBhcmUgYmlnZ2VyIHRoYW4gbWF4aW11bSBvYnNlcnZlZCBjb3VudCBhbmQKbG9uZ2VyIHByZWRpY3RpdmUgZGlzdHJpYnV0aW9uIHRhaWwuIFRoaXMgaXMgbmF0dXJhbCBhcyB1bmNlcnRhaW50eQppbiB0aGUgdmFyaWF0aW9uIGJldHdlZW4gdHVkaWVzIGluY3JlYXNlcyBwcmVkaWN0aXZlIHVuY2VydGFpbnR5LAp0b28sIGVzcGVjaWFsbHkgYXMgdGhlIG51bWJlciBvZiBzdHVkaWVzIGlzIHJlbGF0aXZlbHkgc21hbGwuCgpUaGUgcG9wdWxhdGlvbiBsZXZlbCBjb2VmZmljaWVudCBwb3N0ZXJpb3IgZ2l2ZW4gcG9vbGVkIG1vZGVsCgpgYGB7cn0KcGxvdF9wb3N0ZXJpb3JfcG9vbGVkIDwtIG1jbWNfYXJlYXMoYXNfZHJhd3NfZGYoZml0X3Bvb2xlZCksIHJlZ2V4X3BhcnM9J2JfZG9zZWcnKSArCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0PTAsIGxpbmV0eXBlPSdkYXNoZWQnKSArCiAgbGFicyh0aXRsZT0nUG9vbGVkIG1vZGVsJykKYGBgCgpUaGUgcG9wdWxhdGlvbiBsZXZlbCBjb2VmZmljaWVudCBwb3N0ZXJpb3IgZ2l2ZW4gaGllcmFyY2hpY2FsIG1vZGVsIDEKCmBgYHtyfQpwbG90X3Bvc3Rlcmlvcl9oaWVyMSA8LSBtY21jX2FyZWFzKGFzX2RyYXdzX2RmKGZpdF9oaWVyMSksIHJlZ2V4X3BhcnM9J2JfZG9zZWcnKSArCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0PTAsIGxpbmV0eXBlPSdkYXNoZWQnKSArCiAgbGFicyh0aXRsZT0nSGllcmFyY2hpY2FsIG1vZGVsIDEnKQpgYGAKClRoZSBwb3B1bGF0aW9uIGxldmVsIGNvZWZmaWNpZW50IHBvc3RlcmlvciBnaXZlbiBoaWVyYXJjaGljYWwgbW9kZWwgMwoKYGBge3J9CnBsb3RfcG9zdGVyaW9yX2hpZXIyIDwtIG1jbWNfYXJlYXMoYXNfZHJhd3NfZGYoZml0X2hpZXIyKSwgcmVnZXhfcGFycz0nYl9kb3NlZycpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQ9MCwgbGluZXR5cGU9J2Rhc2hlZCcpICsKICBsYWJzKHRpdGxlPSdIaWVyYXJjaGljYWwgbW9kZWwgMicpCgoocGxvdF9wb3N0ZXJpb3JfcG9vbGVkIC8gcGxvdF9wb3N0ZXJpb3JfaGllcjEgLyBwbG90X3Bvc3Rlcmlvcl9oaWVyMikgKiB4bGltKGMoMCw4LjUpKQpgYGAKCkFsbCBtb2RlbHMgYWdyZWUgdGhhdCB0aGUgc2xvcGUgaXMgdmVyeSBsaWtlbHkgcG9zaXRpdmUuIFRoZQpoaWVyYXJjaGljYWwgbW9kZWxzIGhhdmUgbW9yZSB1bmNlcnRhaW50eSwgYnV0IGFsc28gaGlnaGVyCnBvc3RlcmlvciBtZWFuLgoKV2hlbiB3ZSBsb29rIGF0IHRoZSBzdHVkeSBzcGVjaWZpYyBwYXJhbWV0ZXJzLCB3ZSBzZWUgdGhhdCB0aGUKTWlsbGVyIHN0dWR5IGhhcyBzbGlnaHRseSBoaWdoZXIgaW50ZXJjZXB0IChsZWFkaW5nIHRvIGhpZ2hlciB0aGV0YSkuCgpgYGB7cn0KKG1jbWNfYXJlYXMoYXNfZHJhd3NfZGYoZml0X2hpZXIxKSwgcmVnZXhfcGFycz0ncl9zdHVkeVxcWy4qSW50ZXJjZXB0JykgKwogICBsYWJzKHRpdGxlPSdIaWVyYXJjaGljYWwgbW9kZWwgMScpKSAvCiAgKG1jbWNfYXJlYXMoYXNfZHJhd3NfZGYoZml0X2hpZXIyKSwgcmVnZXhfcGFycz0ncl9zdHVkeVxcWy4qSW50ZXJjZXB0JykgKwogICAgIGxhYnModGl0bGU9J0hpZXJhcmNoaWNhbCBtb2RlbCAyJykpCmBgYAoKClRoZXJlIGFyZSBubyBjbGVhciBkaWZmZXJlbmNlcyBpbiBzbG9wZXMuCgpgYGB7cn0KbWNtY19hcmVhcyhhc19kcmF3c19kZihmaXRfaGllcjIpLCByZWdleF9wYXJzPSdyX3N0dWR5XFxbLipkb3NlZycpICsKICBsYWJzKHRpdGxlPSdIaWVyYXJjaGljYWwgbW9kZWwgMicpCmBgYAoKCkJhc2VkIG9uIExPTyBjb21wYXJpc29uIHdlIGNvdWxkIGNvbnRpbnVlIHdpdGggYW55IG9mIHRoZSBtb2RlbHMsCmJ1dCBpZiB3ZSB3YW50IHRvIHRha2UgaW50byBhY2NvdW50IHRoZSB1bmtub3duIHBvc3NpYmxlIHN0dWR5CnZhcmlhdGlvbnMsIGl0IGlzIGJlc3QgdG8gY29udGludWUgd2l0aCB0aGUgaGllcmFyY2hpY2FsIG1vZGVsIDIuCldlIGNvdWxkIHJlZHVjZSB0aGUgdW5jZXJ0YWludHkgYnkgc3BlbmRpbmcgc29tZSBlZmZvcnQgdG8gZWxpY2l0IGEKbW9yZSBpbmZvcm1hdGl2ZSBwcmlvcnMgZm9yIHRoZSBiZXR3ZWVuIHN0dWR5IHZhcmlhdGlvbiwgYnkKc2VhcmNoaW5nIG9wZW4gc3R1ZHkgZGF0YWJzZXMgZm9yIHNpbWlsYXIgc3R1ZGllcy4gSW4gdGhpcyBleGFtcGxlLAp3ZSBza2lwIHRoYXQgYW5kIGNvbnRpbnVlIHdpdGggb3RoZXIgcGFydHMgb2YgdGhlIHdvcmtmbG93LgoKCmBgYHtyfQpmaXRfaGllcjIgfD4KICBwb3dlcnNjYWxlX3NlcXVlbmNlKCkgfD4KICBwb3dlcnNjYWxlX3Bsb3RfZGVucyh2YXJpYWJsZXM9J2JfZG9zZWcnKSArCiAgIyBzd2l0Y2ggcm93cyBhbmQgY29scwogIGZhY2V0X2dyaWQocm93cz12YXJzKC5kYXRhJHZhcmlhYmxlKSwKICAgICAgICAgICAgIGNvbHM9dmFycyguZGF0YSRjb21wb25lbnQpKSArCiAgIyBjbGVhbmluZwogIGdndGl0bGUoTlVMTCxOVUxMKSArCiAgbGFicyh4PSdEb3NlIChnKSBjb2VmZmljaWVudCcsIHk9TlVMTCkgKwogIHNjYWxlX3lfY29udGludW91cyhicmVha3M9TlVMTCkgKwogIHRoZW1lKGF4aXMubGluZS55PWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBzdHJpcC50ZXh0Lnk9ZWxlbWVudF9ibGFuaygpKQpgYGAKClN1bW1hcmlzZSB0aGUgcHJpb3IgYW5kIGxpa2VsaWhvb2Qgc2Vuc2l0aXZpdHkgdXNpbmcgY3VtdWxhdGl2ZQpKZW5zZW4tU2hhbm5vbiBkaXN0YW5jZSBmb2N1c2luZyBvbiB0aGUgY29tbW9uIHBvcHVsYXRpb24gbGV2ZWwKaW50ZXJjZXB0LgoKYGBge3J9CnBvd2Vyc2NhbGVfc2Vuc2l0aXZpdHkoZml0X2hpZXIyLCB2YXJpYWJsZT0nYl9kb3NlZycKICAgICAgICAgICAgICAgICAgICAgICApJHNlbnNpdGl2aXR5IHw+CiAgICAgICAgICAgICAgICAgICAgICAgICBtdXRhdGUoYWNyb3NzKHdoZXJlKGlzLmRvdWJsZSksICB+bnVtKC54LCBkaWdpdHM9MikpKQpgYGAKClRoZSBwb3N0ZXJpb3IgZm9yIHRoZSBwcm9iYWJpbGl0eSBvZiBldmVudCBnaXZlbiBjZXJ0YWluIGRvc2UgYW5kIGEKbmV3IHN0dWR5IGZvciBoaWVyYXJjaGljYWwgbW9kZWwgMi4KCmBgYHtyfQpkYXRhLmZyYW1lKHN0dWR5PSduZXcnLAogICAgICAgICAgIGRvc2VnPXNlcSgwLjEsMSxieT0wLjEpLAogICAgICAgICAgIHRvdGFsPTEpIHw+CiAgYWRkX2xpbnByZWRfZHJhd3MoZml0X2hpZXIyLCB0cmFuc2Zvcm09VFJVRSwgYWxsb3dfbmV3X2xldmVscz1UUlVFKSB8PgogIGdncGxvdChhZXMoeD1kb3NlZywgeT0ubGlucHJlZCkpICsKICBzdGF0X2xpbmVyaWJib24oLndpZHRoID0gYyguOTUpLCBhbHBoYSA9IDEvMiwgY29sb3I9YnJld2VyLnBhbCg1LCAiQmx1ZXMiKVtbNV1dKSArCiAgc2NhbGVfZmlsbF9icmV3ZXIoKSsKICBsYWJzKHg9ICJEb3NlIChnKSIsIHkgPSAnUHJvYmFiaWxpdHkgb2YgZXZlbnQnLCB0aXRsZT0nSGllcmFyY2hpY2FsIG1vZGVsJykgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpICsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQ9MCkgKwogIHNjYWxlX3hfY29udGludW91cyhicmVha3M9c2VxKDAuMSwxLGJ5PTAuMSkpICsKICB5bGltKGMoMCwwLjE1KSkKYGBgCgpJZiB3ZSBwbG90IGluZGl2aWR1YWwgcG9zdGVyaW9yIGRyYXdzLCB3ZSBzZWUgdGhhdCB0aGVyZSBpcyBhIGxvdCBvZgp1bmNlcnRhaW50eSBhYm91dCB0aGUgb3ZlcmFsbCBwcm9iYWJpbGl0eSAoZXhwbGFpbmVkIGJ5IHRoZQp2YXJpYXRpb24gaW4gSW50ZXJjZXB0IGluIGRpZmZlcmVudCBzdHVkaWVzKSwgYnV0IGxlc3MgdW5jZXJ0YWludHkKYWJvdXQgdGhlIHNsb3BlLgoKYGBge3J9CmRhdGEuZnJhbWUoc3R1ZHk9J25ldycsCiAgICAgICAgICAgZG9zZWc9c2VxKDAuMSwxLGJ5PTAuMSksCiAgICAgICAgICAgdG90YWw9MSkgfD4KICBhZGRfbGlucHJlZF9kcmF3cyhmaXRfaGllcjIsIHRyYW5zZm9ybT1UUlVFLCBhbGxvd19uZXdfbGV2ZWxzPVRSVUUsIG5kcmF3cz0xMDApIHw+CiAgZ2dwbG90KGFlcyh4PWRvc2VnLCB5PS5saW5wcmVkKSkgKwogIGdlb21fbGluZShhZXMoZ3JvdXA9LmRyYXcpLCBhbHBoYSA9IDEvMiwgY29sb3IgPSBicmV3ZXIucGFsKDUsICJCbHVlcyIpW1szXV0pKwogIHNjYWxlX2ZpbGxfYnJld2VyKCkrCiAgbGFicyh4PSAiRG9zZSAoZykiLCB5ID0gJ1Byb2JhYmlsaXR5IG9mIGV2ZW50JykgKwogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpICsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQ9MCkgKwogIHNjYWxlX3hfY29udGludW91cyhicmVha3M9c2VxKDAuMSwxLGJ5PTAuMSkpCmBgYAoKCiMgSGllcmFyY2hpY2FsIGJpbm9taWFsIG1vZGVsIDIKCltTdHVkaWVzIG9uIFBoYXJtYWNvbG9naWMgVHJlYXRtZW50cyBmb3IgQ2hyb25pYyBPYnN0cnVjdGl2ZQpQdWxtb25hcnkKRGlzZWFzZV0oaHR0cHM6Ly93dmllY2h0Yi5naXRodWIuaW8vbWV0YWRhdC9yZWZlcmVuY2UvZGF0LmJha2VyMjAwOS5odG1sKQppbmNsdWRlcyByZXN1bHRzIGZyb20gMzkgdHJpYWxzIGV4YW1pbmluZyBwaGFybWFjb2xvZ2ljIHRyZWF0bWVudHMKZm9yIGNocm9uaWMgb2JzdHJ1Y3RpdmUgcHVsbW9uYXJ5IGRpc2Vhc2UgKENPUEQpLgoKTG9hZCBkYXRhCgpgYGB7cn0KbG9hZCh1cmwoJ2h0dHBzOi8vZ2l0aHViLmNvbS93dmllY2h0Yi9tZXRhZGF0L3Jhdy9tYXN0ZXIvZGF0YS9kYXQuYmFrZXIyMDA5LnJkYScpKQojIGZvcmNlIGNoYXJhY3RlciBzdHJpbmdzIHRvIGZhY3RvcnMgZm9yIGVhc2llciBwbG90aW5nCmRhdC5iYWtlcjIwMDkgPC0gZGF0LmJha2VyMjAwOSB8PgogIG11dGF0ZShzdHVkeSA9IGZhY3RvcihzdHVkeSksCiAgICAgICAgIHRyZWF0bWVudCA9IGZhY3Rvcih0cmVhdG1lbnQpLAogICAgICAgICBpZCA9IGZhY3RvcihpZCkpCmBgYAoKTG9vayBhdCBzaXggZmlyc3QgbGluZXMgb2YgdGhlIGRhdGEgZnJhbWUKCmBgYHtyfQpoZWFkKGRhdC5iYWtlcjIwMDkpCmBgYAoKVG90YWwgbnVtYmVyIG9mIHBhdGllbnRzIGluIGVhY2ggc3R1ZHkgdmFyaWVzIGEgbG90CgpgYGB7cn0KZGF0LmJha2VyMjAwOSB8PgogIGdyb3VwX2J5KHN0dWR5KSB8PgogIHN1bW1hcmlzZShOID0gc3VtKHRvdGFsKSkgfD4KICBnZ3Bsb3QoYWVzKHg9TiwgeT1zdHVkeSkpICsKICBnZW9tX2NvbChmaWxsPTQpICsKICBsYWJzKHg9J051bWJlciBvZiBwYXRpZW50cyBwZXIgc3R1ZHknLCB5PSdTdHVkeScpCmBgYAoKTm9uZSBvZiB0aGUgdHJlYXRtZW50cyBpcyBpbmNsdWRlZCBpbiBldmVyeSBzdHVkeSwgYW5kIGVhY2ggc3R1ZHkgaW5jbHVkZXMKJDItLTQkIHRyZWF0bWVudHMuCgpgYGB7cn0KY3Jvc3N0YWIgPC0gd2l0aChkYXQuYmFrZXIyMDA5LHRhYmxlKHN0dWR5LCB0cmVhdG1lbnQpKQojCnBsb3RfdHJlYXRtZW50cyA8LSBkYXRhLmZyYW1lKG51bWJlcl9vZl9zdHVkaWVzPWNvbFN1bXMoY3Jvc3N0YWIpLCB0cmVhdG1lbnQ9Y29sbmFtZXMoY3Jvc3N0YWIpKSB8PgogIGdncGxvdChhZXMoeD1udW1iZXJfb2Zfc3R1ZGllcyx5PXRyZWF0bWVudCkpICsKICBnZW9tX2NvbChmaWxsPTQpICsKICBsYWJzKHg9J051bWJlciBvZiBzdHVkaWVzIHdpdGggYSB0cmVhdG1lbnQgWCcsIHk9J1RyZWF0bWVudCcpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQ9bnJvdyhjcm9zc3RhYiksIGxpbmV0eXBlPSdkYXNoZWQnKSArCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcz1jKDAsMTAsMjAsMzAsMzkpKQojCnBsb3Rfc3R1ZGllcyA8LSBkYXRhLmZyYW1lKG51bWJlcl9vZl90cmVhdG1lbnRzPXJvd1N1bXMoY3Jvc3N0YWIpLCBzdHVkeT1yb3duYW1lcyhjcm9zc3RhYikpIHw+CiAgZ2dwbG90KGFlcyh4PW51bWJlcl9vZl90cmVhdG1lbnRzLHk9c3R1ZHkpKSArCiAgZ2VvbV9jb2woZmlsbD00KSArCiAgbGFicyh4PSdOdW1iZXIgb2YgdHJlYXRtZW50cyBpbiBhIHN0dWR5IFknLCB5PSdTdHVkeScpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQ9bmNvbChjcm9zc3RhYiksIGxpbmV0eXBlPSdkYXNoZWQnKSArCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcz1jKDAsMiw0LDYsOCkpCiMKcGxvdF90cmVhdG1lbnRzICsgcGxvdF9zdHVkaWVzCmBgYAoKVGhlIGZpcnN0IG1vZGVsIGlzIHBvb2xpbmcgdGhlIGluZm9ybWF0aW9uIG92ZXIgc3R1ZGllcywgYnV0IGVzdGltYXRpbmcgc2VwYXJhdGUKdGhldGEgZm9yIGVhY2ggdHJlYXRtZW50IChpbmNsdWRpbmcgcGxhY2VibykuCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X3Bvb2xlZCA8LSBicm0oZXhhYyB8IHRyaWFscyh0b3RhbCkgfiAwICsgdHJlYXRtZW50LAogICAgICAgICAgICAgICAgICBwcmlvciA9IHByaW9yKHN0dWRlbnRfdCg3LCAwLCAxLjUpLCBjbGFzcz0nYicpLAogICAgICAgICAgICAgICAgICBmYW1pbHk9Ymlub21pYWwoKSwgZGF0YT1kYXQuYmFrZXIyMDA5KQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4KCmBgYHtyfQpmaXRfcG9vbGVkCmBgYAoKVHJlYXRtZW50IGVmZmVjdCBwb3N0ZXJpb3JzCgpgYGB7cn0KZml0X3Bvb2xlZCB8PgogIGFzX2RyYXdzX2RmKCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J2JfJywgcmVnZXg9VFJVRSkgfD4KICBzZXRfdmFyaWFibGVzKHBhc3RlMCgnYl90cmVhdG1lbnRbJywgbGV2ZWxzKGZhY3RvcihkYXQuYmFrZXIyMDA5JHRyZWF0bWVudCkpLCAnXScpKSB8PgogIGFzX2RyYXdzX3J2YXJzKCkgfD4KICBzcHJlYWRfcnZhcnMoYl90cmVhdG1lbnRbdHJlYXRtZW50XSkgfD4KICBtdXRhdGUodGhldGFfdHJlYXRtZW50ID0gcmZ1bihwbG9naXMpKGJfdHJlYXRtZW50KSkgfD4KICBnZ3Bsb3QoYWVzKHhkaXN0PXRoZXRhX3RyZWF0bWVudCwgeT10cmVhdG1lbnQpKSArCiAgc3RhdF9oYWxmZXllKCkgKwogIGxhYnMoeD0ndGhldGEnLCB5PSdUcmVhdG1lbnQnLCB0aXRsZT0nUG9vbGVkIG92ZXIgc3R1ZGllcywgc2VwYXJhdGUgb3ZlciB0cmVhdG1lbnRzJykgIApgYGAKClRyZWF0bWVudCBlZmZlY3Qgb2Rkcy1yYXRpbyBwb3N0ZXJpb3JzCgpgYGB7cn0KdGhldGEgPC0gZml0X3Bvb2xlZCB8PgogIGFzX2RyYXdzX2RmKCkgfD4KICBzdWJzZXRfZHJhd3ModmFyaWFibGU9J2JfJywgcmVnZXg9VFJVRSkgfD4KICBzZXRfdmFyaWFibGVzKHBhc3RlMCgnYl90cmVhdG1lbnRbJywgbGV2ZWxzKGZhY3RvcihkYXQuYmFrZXIyMDA5JHRyZWF0bWVudCkpLCAnXScpKSB8PgogIGFzX2RyYXdzX3J2YXJzKCkgfD4KICBzcHJlYWRfcnZhcnMoYl90cmVhdG1lbnRbdHJlYXRtZW50XSkgfD4KICBtdXRhdGUodGhldGFfdHJlYXRtZW50ID0gcmZ1bihwbG9naXMpKGJfdHJlYXRtZW50KSkKdGhldGFfcGxhY2VibyA8LSBmaWx0ZXIodGhldGEsdHJlYXRtZW50PT0nUGxhY2VibycpJHRoZXRhX3RyZWF0bWVudFtbMV1dCnRoZXRhIHw+CiAgbXV0YXRlKHRyZWF0bWVudF9vZGRzcmF0aW8gPSAodGhldGFfdHJlYXRtZW50LygxLXRoZXRhX3RyZWF0bWVudCkpLyh0aGV0YV9wbGFjZWJvLygxLXRoZXRhX3BsYWNlYm8pKSkgfD4KICBmaWx0ZXIodHJlYXRtZW50ICE9ICJQbGFjZWJvIikgfD4KICBnZ3Bsb3QoYWVzKHhkaXN0PXRyZWF0bWVudF9vZGRzcmF0aW8sIHk9dHJlYXRtZW50KSkgKwogIHN0YXRfaGFsZmV5ZSgpICsKICBsYWJzKHg9J09kZHMtcmF0aW8nLCB5PSdUcmVhdG1lbnQnLCB0aXRsZT0nUG9vbGVkIG92ZXIgc3R1ZGllcywgc2VwYXJhdGUgb3ZlciB0cmVhdG1lbnRzJykgKwogIGdlb21fdmxpbmUoeGludGVyY2VwdD0xLCBsaW5ldHlwZT0nZGFzaGVkJykKYGBgCgpXZSBzZWUgYSBiaWcgdmFyaWF0aW9uIGJldHdlZW4gdHJlYXRtZW50cyBhbmQgdHdvIHRyZWF0bWVudHMgc2VlbQp0byBiZSBoYXJtZnVsLCB3aGljaCBpcyBzdXNwaWNpb3VzLiBMb29raW5nIGF0IHRoZSBkYXRhIHdlIHNlZSB0aGF0Cm5vdCBhbGwgc3R1ZGllcyBpbmNsdWRlZCBhbGwgdHJlYXRtZW50cywgYW5kIHRodXMgaWYgc29tZSBvZiB0aGUKc3R1ZGllcyBoYWQgbW9yZSBldmVudHMsIHRoZW4gdGhlIGFib3ZlIGVzdGltYXRlcyBjYW4gYmUgd3JvbmcuCgpUaGUgdGFyZ2V0IGlzIGRpc2NyZXRlIGNvdW50LCBidXQgYXMgdGhlIHJhbmdlIG9mIGNvdW50cyBpcyBiaWcsIGEKcm9vdG9ncmFtIHdvdWxkIGxvb2sgbWVzc3ksIGFuZCBkZW5zaXR5IG92ZXJsYXkgcGxvdCBpcyBhIGJldHRlcgpjaG9pY2UuICBQb3N0ZXJpb3IgcHJlZGljdGl2ZSBjaGVja2luZyB3aXRoIGtlcm5lbCBkZW5zaXR5CmVzdGltYXRlcyBmb3IgdGhlIGRhdGEgYW5kIDEwIHBvc3RlcmlvciBwcmVkaWN0aXZlIHJlcGxpY2F0ZXMgc2hvd3MKY2xlYXIgZGlzY3JlcGFuY3kuCgpgYGB7cn0KcHBfY2hlY2soZml0X3Bvb2xlZCwgdHlwZT0nZGVuc19vdmVybGF5JykKYGBgCgpQb3N0ZXJpb3IgcHJlZGljdGl2ZSBjaGVja2luZyB3aXRoIFBJVCB2YWx1ZXMgYW5kIEVDREYgZGlmZmVyZW5jZQpwbG90IHdpdGggZW52ZWxvcGUgc2hvd3MgY2xlYXIgZGlzY3JlcGFuY3kuCgpgYGB7cn0KcHBfY2hlY2soZml0X3Bvb2xlZCwgdHlwZT0ncGl0X2VjZGYnLCBuZHJhd3M9NDAwMCkKYGBgCgpQb3N0ZXJpb3IgcHJlZGljdGl2ZSBjaGVja2luZyB3aXRoIExPTy1QSVQgdmFsdWVzIHNob3cgY2xlYXIgZGlzY3JlcGFuY3kuCgpgYGB7cn0KcHBfY2hlY2soZml0X3Bvb2xlZCwgdHlwZT0nbG9vX3BpdF9xcScsIG5kcmF3cz00MDAwKSArCiAgZ2VvbV9hYmxpbmUoKSArCiAgeWxpbShjKDAsMSkpCmBgYAoKVGhlIHNlY29uZCBtb2RlbCB1c2VzIGEgaGllYXJjaGljYWwgbW9kZWwgYm90aCBmb3IgdHJlYXRtZW50CmVmZmVjdHMgYW5kIHN0dWR5IGVmZmVjdHMuCgpgYGB7ciByZXN1bHRzPSdoaWRlJ30KZml0X2hpZXIgPC0gYnJtKGV4YWMgfCB0cmlhbHModG90YWwpIH4gKDEgfCB0cmVhdG1lbnQpICsgKDEgfCBzdHVkeSksCiAgICAgICAgICAgICAgICBmYW1pbHk9Ymlub21pYWwoKSwgZGF0YT1kYXQuYmFrZXIyMDA5KQpgYGAKCkNoZWNrIHRoZSBzdW1tYXJ5IG9mIHRoZSBwb3N0ZXJpb3IgYW5kIGluZmVyZW5jZSBkaWFnbm9zdGljcy4KCmBgYHtyfQpmaXRfaGllcgpgYGAKCkxPTy1DViBjb21wYXJpc29uCgpgYGB7cn0KbG9vX2NvbXBhcmUobG9vKGZpdF9wb29sZWQpLCBsb28oZml0X2hpZXIpKQpgYGAKCldlIGdldCB3YXJuaW5ncyBhYm91dCBQYXJldG8gaydzID4gMC43IGluIFBTSVMtTE9PLCBidXQgYXMgdGhlCmRpZmZlcmVuY2UgYmV0d2VlbiB0aGUgbW9kZWxzIGlzIGh1Z2UsIHdlIGNhbiBiZSBjb25maWRlbnQgdGhhdCB0aGUKb3JkZXIgd291bGQgdGhlIHNhbWUgaWYgd2UgZml4ZWQgdGhlIGNvbXB1dGF0aW9uLCBhbmQgdGhlCmhpZXJhcmNoaWNhbCBtb2RlbCBpcyBtdWNoIGJldHRlciBhbmQgdGhlcmUgaXMgaGlnaCB2YXJpYXRpb24KYmV0d2VlbiBzdHVkaWVzLiBDbGVhcmx5IHRoZXJlIGFyZSBtYW55IGhpZ2hseSBpbmZsdWVudGlhbApvYnNlcnZhdGlvbnMuCgpQb3N0ZXJpb3IgcHJlZGljdGl2ZSBjaGVja2luZyB3aXRoIGtlcm5lbCBkZW5zaXR5IGVzdGltYXRlcyBmb3IgdGhlCmRhdGEgYW5kIDEwIHBvc3RlcmlvciBwcmVkaWN0aXZlIHJlcGxpY2F0ZXMgbG9va3MgZ29vZCAoYWx0aG91Z2gKd2l0aCB0aGlzIG1hbnkgcGFyYW1ldGVycywgdGhpcyBjaGVjayBpcyBsaWtlbHkgdG8gYmUgb3B0aW1pc3RpYykuCgpgYGB7cn0KcHBfY2hlY2soZml0X2hpZXIsIHR5cGU9J2RlbnNfb3ZlcmxheScpCmBgYAoKUG9zdGVyaW9yIHByZWRpY3RpdmUgY2hlY2tpbmcgd2l0aCBQSVQgdmFsdWVzIGFuZCBFQ0RGIGRpZmZlcmVuY2UKcGxvdCB3aXRoIGVudmVsb3BlIGxvb2tzIGdvb2QgKGFsdGhvdWdoIHdpdGggdGhpcyBtYW55IHBhcmFtZXRlcnMsCnRoaXMgY2hlY2sgaXMgbGlrZWx5IHRvIGJlIG9wdGltaXN0aWMpLgoKYGBge3J9CnBwX2NoZWNrKGZpdF9oaWVyLCB0eXBlPSdwaXRfZWNkZicsIG5kcmF3cz00MDAwKQpgYGAKClBvc3RlcmlvciBwcmVkaWN0aXZlIGNoZWNraW5nIHdpdGggTE9PLVBJVCB2YWx1ZXMgbG9vayBnb29kCihhbGhvdWdoIGFzIHRoZXJlIGFyZSBQYXJldG8ta2hhdCB3YXJuaW5ncywgaXQgaXMgcG9zc2libGUgdGhhdAp0aGlzIGRpYWdub3N0aWMgaXMgb3B0aW1pc3RpYykuCgpgYGB7cn0KcHBfY2hlY2soZml0X2hpZXIsIHR5cGU9J2xvb19waXRfcXEnLCBuZHJhd3M9NDAwMCkgKwogIGdlb21fYWJsaW5lKCkgKwogIHlsaW0oYygwLDEpKQpgYGAKClRyZWF0bWVudCBlZmZlY3QgcG9zdGVyaW9ycyBoYXZlIG5vdyBtdWNoIGxlc3MgdmFyaWF0aW9uLgoKYGBge3J9CmZpdF9oaWVyIHw+CiAgc3ByZWFkX3J2YXJzKGJfSW50ZXJjZXB0LCByX3RyZWF0bWVudFt0cmVhdG1lbnQsXSkgfD4KICBtdXRhdGUodGhldGFfdHJlYXRtZW50ID0gcmZ1bihwbG9naXMpKGJfSW50ZXJjZXB0ICsgcl90cmVhdG1lbnQpKSB8PgogIGdncGxvdChhZXMoeGRpc3Q9dGhldGFfdHJlYXRtZW50LCB5PXRyZWF0bWVudCkpICsKICBzdGF0X2hhbGZleWUoKSArCiAgbGFicyh4PSd0aGV0YScsIHk9J1RyZWF0bWVudCcsIHRpdGxlPSdIaWVyYXJjaGljYWwgb3ZlciBzdHVkaWVzLCBoaWVyYXJjaGljYWwgb3ZlciB0cmVhdG1lbnRzJykgIApgYGAKClN0dWR5IGVmZmVjdCBwb3N0ZXJpb3JzIHNob3cgdGhlIGV4cGVjdGVkIGhpZ2ggdmFyaWF0aW9uLgoKYGBge3J9CmZpdF9oaWVyIHw+CiAgc3ByZWFkX3J2YXJzKGJfSW50ZXJjZXB0LCByX3N0dWR5W3N0dWR5LF0pIHw+CiAgbXV0YXRlKHRoZXRhX3N0dWR5ID0gcmZ1bihwbG9naXMpKGJfSW50ZXJjZXB0ICsgcl9zdHVkeSkpIHw+CiAgZ2dwbG90KGFlcyh4ZGlzdD10aGV0YV9zdHVkeSwgeT1zdHVkeSkpICsKICBzdGF0X2hhbGZleWUoKSArCiAgbGFicyh4PSd0aGV0YScsIHk9J1N0dWR5JywgdGl0bGU9J0hpZXJhcmNoaWNhbCBvdmVyIHN0dWRpZXMsIGhpZXJhcmNoaWNhbCBvdmVyIHRyZWF0bWVudHMnKSAgCmBgYAoKVHJlYXRtZW50IGVmZmVjdCBvZGRzLXJhdGlvIHBvc3RlcmlvcnMKCmBgYHtyfQp0aGV0YSA8LSBmaXRfaGllciB8PgogIHNwcmVhZF9ydmFycyhiX0ludGVyY2VwdCwgcl90cmVhdG1lbnRbdHJlYXRtZW50LF0pIHw+CiAgbXV0YXRlKHRoZXRhX3RyZWF0bWVudCA9IHJmdW4ocGxvZ2lzKShiX0ludGVyY2VwdCArIHJfdHJlYXRtZW50KSkKdGhldGFfcGxhY2VibyA8LSBmaWx0ZXIodGhldGEsdHJlYXRtZW50PT0nUGxhY2VibycpJHRoZXRhX3RyZWF0bWVudFtbMV1dCnRoZXRhIHw+CiAgbXV0YXRlKHRyZWF0bWVudF9vZGRzcmF0aW8gPSAodGhldGFfdHJlYXRtZW50LygxLXRoZXRhX3RyZWF0bWVudCkpLyh0aGV0YV9wbGFjZWJvLygxLXRoZXRhX3BsYWNlYm8pKSkgfD4KICBmaWx0ZXIodHJlYXRtZW50ICE9ICJQbGFjZWJvIikgfD4KICBnZ3Bsb3QoYWVzKHhkaXN0PXRyZWF0bWVudF9vZGRzcmF0aW8sIHk9dHJlYXRtZW50KSkgKwogIHN0YXRfaGFsZmV5ZSgpICsKICBsYWJzKHg9J09kZHMtcmF0aW8nLCB5PSdUcmVhdG1lbnQnLCB0aXRsZT0nSGllcmFyY2hpY2FsIG92ZXIgc3R1ZGllcywgaGllcmFyY2hpY2FsIG92ZXIgdHJlYXRtZW50cycpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQ9MSwgbGluZXR5cGU9J2Rhc2hlZCcpCmBgYAoKVHJlYXRtZW50IGVmZmVjdCBvZGRzLXJhdGlvcyBsb29rIG5vdyBtb3JlIHJlYXNvbmFibGUuIEFzIG5vdyBhbGwKdHJlYXRtZW50cyB3ZXJlIGNvbXBhcmVkIHRvIHBsYWNlYm8sIHRoZXJlIGlzIGxlc3Mgb3ZlcmxhcCBpbiB0aGUKZGlzdHJpYnV0aW9ucyBhcyB3aGVuIGxvb2tpbmcgYXQgdGhlIHRoZXRhcywgYXMgYWxsIHRoZXRhcyBpbmNsdWRlCnNpbWlsYXIgdW5jZXJ0YWludHkgYWJvdXQgdGhlIG92ZXJhbGwgdGhldGEgZHVlIHRvIGhpZ2ggdmFyaWF0aW9uCmJldHdlZW4gc3R1ZGllcy4KVGhlIHRoaXJkIG1vZGVsIGluY2x1ZGVzIGludGVyYWN0aW9uIHNvIHRoYXQgdGhlIHRyZWF0bWVudCBjYW4gZGVwZW5kIG9uIHN0dWR5LgoKYGBge3IgcmVzdWx0cz0naGlkZSd9CmZpdF9oaWVyMiA8LSBicm0oZXhhYyB8IHRyaWFscyh0b3RhbCkgfiAoMSB8IHRyZWF0bWVudCkgKyAodHJlYXRtZW50IHwgc3R1ZHkpLAogICAgICAgICAgICAgICAgZmFtaWx5PWJpbm9taWFsKCksIGRhdGE9ZGF0LmJha2VyMjAwOSwgY29udHJvbD1saXN0KGFkYXB0X2RlbHRhPTAuOSkpCmBgYAoKTE9PIGNvbXBhcmlzb24gc2hvd3MKCmBgYHtyfQpsb29fY29tcGFyZShsb28oZml0X2hpZXIpLCBsb28oZml0X2hpZXIyKSkKYGBgCgpXZSBnZXQgd2FybmluZ3MgYWJvdXQgUGFyZXRvIGsncyA+IDAuNyBpbiBQU0lTLUxPTywgYnV0IGFzIHRoZQptb2RlbHMgYXJlIHNpbWlsYXIsIGFuZCB0aGUgZGlmZmVyZW5jZSBpcyBzbWFsbCwgd2UgY2FuIGJlCnJlbGF0aXZlbHkgY29uZmlkZW50IHRoYXQgdGhlIG1vcmUgY29tcGxleCBtb2RlbCBpcyBub3QgYmV0dGVyLgoKPGJyIC8+CgojIExpY2Vuc2VzIHsudW5udW1iZXJlZH0KCiogQ29kZSAmY29weTsgMjAxNy0yMDI0LCBBa2kgVmVodGFyaSwgbGljZW5zZWQgdW5kZXIgQlNELTMuCiogVGV4dCAmY29weTsgMjAxNy0yMDI0LCBBa2kgVmVodGFyaSwgbGljZW5zZWQgdW5kZXIgQ0MtQlktTkMgNC4wLgo=